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EXECUTIVE  SUMMARY 


Achieving  high  effectiveness  in  project  management  helps  to  ensure  a  successful 
outcome  from  the  project.  Currently,  the  software  engineering  body  of  knowledge  lacks 
adequate  tools  that  will  help  to  quantify  the  project  management  effectiveness  in  software 
projects.  Furthennore,  the  ability  to  assess  or  measure  the  status  is  essential  to  conduct 
any  formal  process  improvement  effort.  In  this  dissertation,  a  software  project 
management  effectiveness  metric  is  introduced.  This  metric  provides  a  standard 
quantitative  measure  of  project  management  effectiveness  from  project  start  to  project 
delivery.  It  will  help  managers  in  software  project  development  organizations  to  evaluate, 
monitor  and  improve  project  management  effectiveness.  Simply,  this  metric  will  guide 
project  stakeholders  to  achieve  better  project  outcomes,  such  as  completing  the  project  on 
time,  within  budget,  with  required  functionality  and  with  creation  of  value  for  project 
stakeholders. 

Twenty  survey  studies  on  software  projects  are  conducted  to  investigate  the 
applicability  and  limitations  of  the  metric.  In  addition,  survey  studies  provided  the 
necessary  empirical  evidence  required  for  the  validation  of  the  metric.  Pearson  product 
moment  correlation  analysis  on  the  data  gathered  from  survey  studies  showed  that  there 
is  a  strong  positive  correlation  with  software  project  success  ratings  provided  by  study 
participants  from  survey  studies  and  project  management  effectiveness  measurements. 
The  result  of  the  analysis  on  the  data  set  indicates  that  half  of  the  variation  in  software 
project  success  may  be  explained  by  the  project  management  effectiveness  metric.  This 
and  other  results  based  on  the  analysis  conducted  on  the  data  set  shows  that  the  proposed 
software  project  management  effectiveness  measurement  is  sound. 

The  measurement  of  software  project  management  effectiveness  involves  the  use 
of  two  new  tools  developed  within  this  research.  The  software  project  management 
evaluation  instrument  (SPMEI)  is  used  to  gather  project  data.  Basically,  this  instrument  is 
a  data  collection  tool  to  gather  project  data  related  to  fifteen  project  management  areas. 
They  are  communication,  teamwork,  leadership,  organizational  commitment,  project 
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manager,  stakeholder  involvement,  staffing  and  hiring,  requirements  management, 
project  planning  and  estimation,  project  monitoring  and  control,  scope  management, 
configuration  management,  quality  engineering,  risk  assessment,  and  risk  control.  The 
instrument  is  comprehensive.  A  member  of  the  project  organization  who  has  broad 
knowledge  on  all  aspects  of  the  project  management  fills  out  this  questionnaire-based 
instrument.  Then  the  data  gathered  with  the  instrument  is  fed  into  the  software  project 
management  evaluation  model  (SPMEM).  This  model  is  used  to  measure  the 
effectiveness  based  on  the  data  gathered  with  the  instrument.  Responses  to  questions  in 
the  instrument  are  assigned  with  specific  scores.  The  evaluation  model  simply  combines 
these  scores  in  a  systematic  way  as  it  is  hypothesized.  SPMEM  produces  a  score  for  each 
project  management  area  and  these  scores  are  then  used  to  compute  a  project 
management  effectiveness  (PME)  score  based  on  a  scale  from  0  to  10.  A  score  of  0 
indicates  the  least  effective  project  management  while  a  score  of  10  indicates  the  most 
effective  project  management.  A  high  PME  score  indicates  a  high  probability  of  project 
success  while  a  low  PME  score  indicates  a  low  probability  of  project  success. 

This  doctoral  research  includes  other  contributions  to  the  software  engineering 
body  of  knowledge.  At  the  beginning  of  the  research,  approaches  for  measuring  the 
project  management  effectiveness  in  software  projects  are  identified.  This  identification 
guided  the  selection  of  a  suitable  measurement  approach.  There  are  four  ways  to  assess  or 
measure  the  project  management  effectiveness.  They  are  subjective  evaluation, 
questionnaire-based  measurement,  metrics-based  measurement,  and  model-based 
measurement.  The  chosen  approach  was  the  questionnaire-based  measurement  since  this 
approach  has  shown  promise  in  an  earlier  study  while  two  other  approaches,  metrics- 
based  measurement  and  model-based  measurement,  have  not  been  used  before.  The  other 
approach,  subjective  evaluation,  was  used  by  study  participants  to  rate  the  success  of 
their  projects.  The  identification  of  approaches  for  measuring  project  management 
effectiveness  will  help  researchers  to  develop  other  metrics  in  their  future  studies. 

Project  management  discipline  suffers  from  the  lack  of  a  theory  of  project 
management.  This  important  issue  has  been  raised  by  various  scholars  in  recent  years. 
These  scholars  indicated  that  not  having  a  theory  of  project  management  poses  challenges 
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for  advancing  the  body  of  knowledge.  This  research  was  also  challenged  due  to  the  lack 
of  a  project  management  theory  that  lays  the  foundation  for  the  development  of  a  project 
management  effectiveness  metric.  During  the  early  phases  of  this  research,  the  necessity 
for  the  development  of  a  theory  of  project  management  became  clear.  Therefore  a  simple, 
yet  powerful,  theory  of  project  management  was  developed.  Simplicity  is  important 
because  project  management  is  complex  by  nature  and  development  of  the  metric  would 
be  more  challenging  if  the  theory  could  not  simplify  the  concepts.  This  theory  aided  the 
metric  development  process  by  providing  a  solid  foundation  to  build  upon.  The  core 
concepts  in  the  theory  of  project  management  are  activities  and  entities.  A  project  is 
simply  the  result  of  a  project  management  function  that  takes  various  activities  and 
entities  as  its  inputs.  The  ideal  goal  of  project  management  is  to  find  the  right  and 
efficient  combination  of  necessary  activities  and  entities  to  reach  the  desired  outcome. 
Identification  of  activities  and  entities  and  arrangement  of  these  activities  and  entities  in 
an  appropriate  order  while  dealing  with  constraints  are  the  tools  to  bring  the  project  to 
life.  The  main  task  in  the  development  of  the  software  project  management  evaluation 
instrument  was  the  identification  of  activities  and  entities  used  in  software  projects. 

A  theory  of  project  management  effectiveness  measurement  is  developed  based 
on  the  theory  of  project  management  and  the  core  concepts  identified  with  it.  The  theory 
of  project  management  effectiveness  measurement  simply  states  that  it  is  possible  to 
measure  project  management  effectiveness  via  measuring  the  effectiveness  of  activities 
and  entities  in  software  projects. 

The  development  of  another  theory  from  the  theory  of  project  management  and 
the  successful  results  achieved  in  this  research  provides  evidence  for  the  validity  of  the 
proposed  theory  of  project  management. 

Software  project  management  is  broad  and  complex  by  nature.  The  theory  of 
project  management  lays  the  foundation  for  further  advancement.  Furthermore,  it  helps 
us  to  overcome  the  challenges  of  dealing  with  this  complexity  by  enabling  us  to 
understand  project  management  in  tenns  of  core  concepts  with  a  simpler  view.  It  is 
essential  to  identify  the  boundaries  as  well.  A  software  project  management  framework 
was  developed  in  this  research  for  two  purposes.  One  of  the  purposes  is  to  identify  the 
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boundaries  of  project  management  and  necessary  project  management  areas  to  be 
included  in  the  effectiveness  measurement.  The  other  one  is  to  categorize  project 
management  areas  with  respect  to  main  areas.  The  main  project  management  areas  are 
people,  process,  product  and  risk.  Each  section  in  SPMEI  corresponds  to  a  project 
management  area  in  the  framework.  Each  project  management  area  is  categorized  under 
one  of  the  main  areas.  The  first  letters  of  these  main  areas  are  used  in  naming  the 
framework.  As  a  result,  it  is  named  as  3PR  framework.  This  framework  guided  the 
development  of  the  metric  by  helping  the  identification  of  what  to  include  in  the 
measurement.  The  3PR  framework  is  simple  and  modifiable.  Therefore,  it  served  well 
during  the  development  of  the  metric.  This  framework  is  validated  via  a  survey  of 
software  practitioners  around  the  world.  In  addition,  this  validation  survey  helped  to 
identify  importance  weightings  for  each  main  project  management  area.  These  weights 
are  used  in  the  development  of  the  software  project  management  evaluation  model  to 
compute  the  effectiveness  score. 

In  this  research,  the  development  of  software  project  management  effectiveness 
metric  required  the  accomplishment  of  a  set  of  related  studies,  mainly  because  either  such 
studies  do  not  exist  in  prior  literature  or  the  existing  literature  are  found  to  be  inadequate. 

The  findings  of  the  analysis  conducted  on  the  data  gathered  from  survey  studies 
indicates  that  the  software  project  management  effectiveness  metric  proposed  in  this 
research  is  sound,  valid  and  applicable  to  be  used  in  software  projects. 
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I.  INTRODUCTION 


A.  BACKGROUND 

The  term  “software  crisis”  was  coined  in  1968  at  a  NATO  working  group  meeting 
on  software  engineering.  At  the  time,  the  term  referred  to  the  inability  of  the  government 
defense  organizations  of  NATO  countries  to  procure  software-intensive  systems  on 
schedule,  within  budget,  and  with  the  desired  level  of  functionality  and  dependability. 
The  discipline  known  as  software  engineering  is  relatively  new  in  relation  to  its  sister 
engineering  disciplines,  but  the  pace  of  the  evolution  of  software  engineering  into  a 
mature  engineering  discipline  needs  to  pick  up  because  the  lion’s  share  of  the 
functionality  in  most  systems  produced  today  is  implemented  in  software. 

A  report  by  the  U.S.  General  Accounting  Office  (GAO)  (1979)  states  that  of  the 
government  software  development  projects  analyzed: 

more  than  50%  had  cost  overruns; 

more  than  60%  had  schedule  overruns; 

more  than  45%  of  the  delivered  software  could  not  be  used; 

more  than  29%  of  the  software  contracted  for  was  never  delivered;  and 

more  than  19%  of  the  software  had  to  be  reworked. 

The  Standish  Group  Report  (1995)  found  that,  on  average,  approximately  16%  of 
software  projects  were  completed  on  time  and  within  budget.  In  addition,  the  projects 
completed  contained  only  approximately  42%  of  the  originally  proposed  features  and 
functions.  These  two  reports  indicate  that  the  situation  has  not  changed. 

The  1987,  the  Defense  Science  Board  reported  that: 

After  two  decades  of  largely  unfulfdled  promises  about  productivity  and 
quality  gains  from  applying  new  software  methodologies  and 
technologies,  industry  and  government  organizations  are  realizing  that 
their  fundamental  problem  is  the  inability  to  manage  the  software 
processes. 
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The  U.S.  DoD  responded  to  this  and  various  statements  in  similar  reports  by 
sponsoring  the  creation  of  the  Software  Engineering  Institute  (SEI)  at  the  Carnegie 
Mellon  University.  SEI  was  tasked  with  finding  ways  to  improve  the  software 
engineering  processes  used  by  the  DoD  and  its  contractors.  One  of  the  improvements 
developed  by  SEI  is  the  Capability  Maturity  Model  (CMM),  which  is  now  in  use 
throughout  the  DoD.  The  CMM  serves  as  a  means  to  identify  key  practices  to  improve 
organizations’  software  development  processes  and  propose  models  to  encompass 
systematic  advancements  in  various  aspects  of  the  process.  The  SEI  specialized  the 
original  model  to  address  software  (SW-CMM),  management  of  human-resources  (P- 
CMM),  systems  engineering  (SE-CMM),  integrated  product  development  (IPD-CMM), 
and  software  acquisition  (SA-CMM).  These  models,  except  for  SA-CMM,  are  now  part 
of  the  Capability  Maturity  Model  Integration  (CMMI). 

The  Algorithmic  Constructive  Cost  Model  (COCOMO),  developed  in  1981,  is 
one  of  the  earliest  and  most  widely  used  software  project  cost-estimation  models 
(Boehm,  1981).  The  model  is  primarily  used  for  developing  predictions  based  on  basic, 
intennediate,  and  detailed  COCOMO;  each  level  provides  a  different  degree  of  rigor. 
Basic  COCOMO  is  only  useful  when  a  quick,  early,  rough  estimate  is  required. 
Intermediate  COCOMO  produces  better  results.  Intermediate  and  detailed  COCOMO 
take  into  account  attributes  of  the  software  product,  computer  hardware,  development 
personnel,  and  the  project.  Detailed  COCOMO  is  more  rigorous  and  describes  the 
software  as  a  module-subsystem-system  hierarchy.  In  this  model,  software  development 
is  estimated  by  phase.  However,  neither  of  these  models  takes  into  account  project 
management  quality.  Boehm,  developer  of  these  models,  indicates  that  poor  management 
can  increase  software  costs  more  rapidly  than  any  other  factor.  Furthermore,  when 
estimating  cost  with  COCOMO,  there  is  an  assumption  built  into  the  model  that  the 
project  will  be  well-managed.  Boehm  also  points  out  that  management  quality  ratings  are 
not  easy  to  determine. 

The  first  version  of  COCOMO  II  was  released  to  the  public  in  1997.  A  few  other 
versions  with  improvements  were  made  available  in  subsequent  years.  COCOMO  II  was 
developed  in  response  to  the  increasing  difficulties  in  cost  estimations  with  COCOMO  8 1 
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and  COCOMO  87.  Accidental  advancements  in  the  field  such  as  development  of  new  life 
cycle  models,  COTS  and  other  reuse  approaches,  object  oriented  approaches,  etc.,  were 
the  reason  for  such  difficulties  (Brooks,  1995).  COCOMO  II  has  three  submodels  called 
Applications  Composition,  Early  Design,  and  Post-architecture  models  (Boehm  et  ah, 
2000).  It  is  crucial  to  state  that  COCOMO  II  recognizes  the  importance  of  maturity  in 
cost,  effort  and  schedule  estimation.  The  model  has  a  parameter  called  “process  maturity” 
as  an  exponent  scale  factor.  Process  maturity  affects  the  estimated  effort  exponentially 
(Boehm  et  ah,  1995). 

Development  of  CMMI  and  COCOMO  models  are  two  important  studies  in  the 
field  of  software  project  management.  These  studies  may  be  complemented  with  the 
introduction  of  a  well-established  software  project  management  effectiveness  metric. 
Such  a  metric  would  complement  the  CMMI  and  existing  metrics,  possibly  contributing 
to  further  improvements  in  the  software  development  processes  and  their  applications  in 
the  development  of  software-intensive  systems.  The  metric  may  be  used  as  an  input  in 
software  cost  estimation  models  for  better  estimates. 

B.  CONTRIBUTIONS  OF  THIS  RESEARCH 

There  is  limited  scientific  work  that  addresses  theories  and  foundations  in 
software  project  management.  The  bulk  of  the  work  related  to  this  area  is  based  on 
experience  reports  with  limited  empirical  studies. 

This  study  expands  the  body  of  knowledge  by  laying  new  foundations  and  an 
introduction  of  a  metric  to  evaluate  the  project  management  effectiveness  in  software 
projects.  The  contributions  are  listed  below: 

1 .  Introduction  of  a  theory  of  project  management. 

2.  Introduction  of  a  theory  of  project  management  effectiveness 
measurement. 

3.  Identification  of  approaches  for  measuring  project  management 
effectiveness. 

4.  Introduction  of  a  software  project  management  framework. 
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5.  Development  of  a  self-evaluation  instrument  for  software  project 
management. 

6.  Development  of  a  metric  for  software  project  management  effectiveness. 

Quality  research  enables  us  to  ask  new  questions  while  providing  answers  to  the 
old  ones.  These  questions  help  us  to  improve  the  field.  This  research  helps  us  to  ask  new 
questions  in  the  field  of  software  project  management. 

C.  OVERVIEW  OF  THE  DISSERTATION 

This  dissertation  consists  of  ten  chapters.  Chapter  I  includes  the  introduction  of 
the  research.  In  the  introduction  section,  statement  and  significance  of  the  problem,  the 
research  hypothesis,  the  research  overview,  the  assumptions  and  applicability  of  the 
metric,  and  the  key  definition  are  presented. 

Chapter  II  discusses  the  related  work.  In  this  chapter,  discussion  of  project 
success,  project  success  and  failure  factors,  the  role  of  project  management  in  achieving 
project  success,  measurement  of  project  success,  and  software  project  management 
effectiveness  are  presented. 

In  Chapter  III,  the  approaches  for  measuring  the  management  effectiveness  of 
software  projects  are  outlined.  Four  different  approaches  are  discussed  with  examples 
from  prior  research  studies.  The  contribution  in  this  chapter  is  the  guidance  for  future 
researchers  in  the  selection  of  metric  development  approaches. 

Chapter  IV  introduces  a  theory  of  project  management.  This  theory  builds  the 
necessary  foundation  for  the  development  of  the  metric.  In  addition,  a  theoretical 
foundation  for  measurement  of  project  management  effectiveness  is  presented. 

A  framework  for  software  project  management  is  introduced  in  Chapter  V. 
Related  frameworks  from  various  standards  and  software  project  management  literature 
are  discussed.  The  project  management  areas  in  the  framework  are  explained. 

In  Chapter  VI,  the  results  from  a  survey  study  on  software  project  management 
are  presented.  The  goal  of  this  survey  study  was  to  validate  the  framework  introduced  in 

the  previous  chapter.  The  results  of  the  survey  supported  the  validation  of  the  framework. 
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In  Chapter  VII,  the  software  project  management  evaluation  instrument  (SPMEI) 
and  the  evaluation  model  are  explained.  The  overall  structure,  important  characteristics, 
and  the  development  of  the  instrument  are  discussed. 

The  analysis  of  the  survey  studies  on  software  projects  can  be  found  in  Chapter 
VIII.  Each  study  is  discussed  briefly.  The  data  analysis  from  these  studies  is  also 
presented  here. 

The  findings  of  this  research  are  presented  in  Chapter  IX,  while  conclusions  and 
future  work  are  discussed  in  the  tenth  and  final  chapter  of  the  dissertation. 

D.  STATEMENT  OF  THE  PROBLEM 

Brooks  (1995)  points  out  in  his  well-known  book  that  there  are  essential  and 
accidental  difficulties  of  software  engineering.  The  accidental  difficulties  are  timely 
problems  of  the  field;  they  are  solved  with  major  breakthroughs  and  solutions,  leading  to 
increases  in  software  development  productivity.  However,  they  are  not  inherent  in  the 
essence  of  software.  According  to  Brooks,  essential  difficulties  are  inherent  to  the  nature 
of  software  and  they  are  complexity,  conformity,  changeability,  and  invisibility.  If  that  is 
the  case,  no  matter  how  much  automation  is  acquired,  software  project  development  will 
continue  to  be  a  human- intensive  task.  Effective  software  project  management  is  an 
important  factor  in  the  success  of  a  software  development  project. 

Without  the  use  of  metrics,  software  engineering  processes  will  continue  to  be  ad 
hoc  processes  at  best.  The  SEI  recognized  this  fact  and  incorporated  a  rating  system  to 
describe  the  maturity  of  an  organization,  from  CMM  level-one  as  ad  hoc  to  level-five  as 
optimizing.  In  order  to  comply  with  CMM  level-four,  organizations  have  to  collect 
metrics  related  to  software  development  processes.  CMM  level-five  necessitates 
continuous  effort  on  gathering  these  metrics  and  applying  the  metrics  to  continuously 
improve  the  process. 

Currently,  the  software  engineering  field  lacks  a  well-founded  software  project 
management  metric.  Such  a  metric  could  enable  software  project  managers  to  measure 
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the  project  management  effectiveness,  identify  problematic  areas  during  projects,  identify 
challenged  areas  in  completed  projects  (e.g.,  postmortem  analysis),  and  shed  light  on 
forthcoming  ones. 

E.  SIGNIFICANCE  OF  THE  PROBLEM 

In  2000,  the  Defense  Science  Board  published  a  report  on  the  subject  of  software 
process,  stating  that: 

. . .  [the]  DSB  Task  Force  observed  that  requirements  and  management  are 
the  hardest  part  of  the  software  task  and  advocated  the  use  of 
revolutionary  practices.  This  is  still  true  today. 

Furthermore,  in  the  major  findings  and  recommendations  section  of  the  same 
report,  that  the  DSB  concluded  that  “In  general,  the  technical  issues,  although  difficult  at 
times,  were  not  the  detennining  factor.  Disciplined  execution  was.” 

DeMarco  and  Lister  state  in  their  recognized  book  about  productive  projects  and 
teams:  “For  overwhelming  majority  of  the  bankrupt  projects  we  studied,  there  was  not  a 
single  technological  issue  to  explain  the  failure.”  Furthermore,  “the  major  problems  of 
our  work  are  not  so  much  technological  as  sociological  in  nature”  (DeMarco  &  Lister, 
1987;  DeMarco  &  Lister,  1999). 

Robertson  and  Robertson  (2005)  start  one  of  the  chapters  named  “Project 
Sociology”  in  their  book  with: 

In  several  decades  of  project  experience,  we  have  never  seen  a  project  fail 
for  technical  reasons.  It  has  always  been  human  failures  that  have  caused 
otherwise  good  projects  to  grind  to  a  halt. 

These  findings  clearly  point  out  that  managing  the  software  development  process 
is  still  a  fundamental  problem  within  the  defense  community  and  the  commercial  world. 
However,  little  research  has  been  devoted  to  this  issue.  In  this  dissertation,  the  goal  is  to 
develop  and  experiment  with  a  project  software  management  metric  to  measure  the 
effectiveness  of  software  project  management. 
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The  benefits  of  gathering  metrics  have  been  advocated  by  many  researchers  and 
practitioners  in  the  field.  Putnam  and  Myers  (2003)  explain  how  to  use  the  metrics  in 
software  projects  and  how  they  are  related  with  each  other.  They  clearly  emphasize  that 
metrics  provide  and  enable: 

•  dependable  estimates  of  project  effort,  schedule,  and  reliability; 

•  control  of  the  project  during  its  course; 

•  ability  to  re-plan  an  errant  project  along  the  way; 

•  master-planning  the  assignment  of  resources  to  all  projects  within  the 
organization; 

•  monitoring  process  improvement  from  year  to  year. 

Having  a  software  project  management  metric  will  complement  the  current  set  of 
metrics  and  provide  us  with  a  broader  understanding  of  software  development  dynamics. 

The  ten  most  important  success  factors  are  identified  in  an  IT  project  by  The 
Standish  Group’s  CHAOS  study  (1994).  In  2000,  the  ratings  of  the  factors  are  updated  by 
The  Standish  Group  (2000).  According  to  the  new  rating,  the  factors,  executive  support 
and  experienced  project  manager,  were  rated  first  and  third  with  a  success  factor  of  18 
and  14  out  of  100  respectively.  A  software  project  management  metric  will  provide  a 
project  management  evaluation  that  can  be  used  by  project  managers  to  better  manage 
software  development  efforts. 

Finally,  Peter  W.  G.  Morris  (Pinto,  1998)  states  that: 

One  of  the  major  areas  of  project  management  development  over  the  next 

few  years,  I  believe,  will  be  establishing  and  refining  interindustry  metrics 

for  quantifying  performance  improvements.  Much  of  this  work  will  be  IT 

related. 

Martha  Gray  (1999)  discusses  the  state  of  software  metrics  and  emphasizes  the 
immaturity  of  software  metrology  and  its  fundamentals.  She  points  out  the  importance  of 
having  measures  for  software  and  IT  industry.  This  study  will  be  an  addition  to  the  set  of 
metrics  and  it  will  fonn  a  basis  for  discussion  over  the  measurement  of  project 
management  effectiveness  in  software  projects. 
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F.  RESEARCH  HYPOTHESIS 

There  is  only  one  hypothesis  in  this  research: 

•  The  success  of  a  software  project  positively  correlates  to  its  project 
management  effectiveness. 

The  success  rating  of  the  project  will  be  provided  by  the  survey  study  participants. 
They  will  assess  the  success  of  a  project  based  on  their  perspectives.  They  will  rate  the 
project  success  on  a  0  to  10  scale,  with  0  being  complete  failure  and  10  being  complete 
success.  The  software  project  management  effectiveness  (PME)  will  be  acquired  using 
the  software  project  management  evaluation  instrument  and  model.  The  PME  metric  will 
be  on  a  scale  from  0  to  10. 

The  testing  of  the  hypothesis  will  be  conducted  by  analyzing  the  Pearson  product- 
moment  correlation  coefficient  (PMCC)  between  these  two  measures. 

G.  RESEARCH  OVERVIEW 

1.  Research  Questions 

We  believe  that  it  is  possible  to  develop  a  software  project  management 
effectiveness  metric.  This  metric  should  be  meaningful  enough  that  it  provides  insights 
on  the  quality  of  the  project  management.  The  measure  should  also  be  able  to  distinguish 
between  the  projects  following  certain  expected  best  practices  and  others  lacking  some  of 
these  practices. 

The  following  questions  will  be  addressed  with  this  research: 

1 .  What  are  the  most  important  project  management  areas? 

2.  What  are  the  possible  approaches  for  measuring  the  project 
management  effectiveness? 

3.  Is  it  possible  to  develop  a  simple  theory  of  project  management  and 
measurement  of  project  management  effectiveness? 
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4.  Is  it  possible  to  develop  a  software  project  management  metric  that 
measures  the  project  management  effectiveness  in  a  software  project? 

5.  Will  this  metric  be  sound,  meaningful  and  applicable? 

Soundness  of  the  metric  is  that  software  practitioners  respond  positively  when 
applying  the  metric.  To  get  some  idea  regarding  the  soundness,  we  can  simply  ask  them 
if  they  find  the  metric  sound  during  its  application.  Further  analysis  can  be  conducted  by 
identifying  and  responding  to  validity  concerns.  Meaningfulness  of  the  metric  is  that  it 
yields  different  results  for  the  project  management  in  which  one  of  them  clearly  lacks 
certain  best  practices.  Applicability  of  the  metric  is  that  the  measurement  is  practical. 

Answering  the  fifth  research  question  in  detail  requires  substantial  research  that 
expands  beyond  the  scope  of  this  dissertation.  However,  some  evidence  can  be  provided 
as  to  whether  such  qualities  are  captured  to  a  certain  level  with  the  metric.  It  is  important 
to  address  these  qualities  in  a  metric  development  effort. 

2.  Research  Strategy 

The  objective  of  this  research  is  to  develop  a  software  project  management 
effectiveness  metric.  The  literature  on  the  subject  is  limited  and  mostly  consists  of  rough 
models  such  as  Pinto  and  Slevin  (1987)  and  Wohlin  and  von  Mayrhauser  (2001)  for 
project  success. 

Identification  of  crucial  and  common  aspects  for  software  project  management  is 
the  first  step  for  this  research.  Second,  this  identification  will  help  us  to  develop  a 
framework  to  work  on.  The  validation  of  the  framework  is  necessary  to  ensure  the 
soundness  of  the  metric.  A  survey  of  software  practitioners  will  be  conducted  for  the 
validation  as  a  third  step.  Then,  using  the  framework  and  its  components,  a  measurement 
tool  will  be  developed.  Finally,  data  will  be  collected  from  real-world  software  projects 
and  the  results  will  be  analyzed. 

Any  meaningful  measurement  instrument  should  be  able  to  distinguish  two 
different  entities  based  on  the  goal  of  the  measurement.  In  addition,  it  is  imperative  to  be 
able  to  substantiate  one  measure  with  another  measure.  Therefore,  in  this  research,  two 
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different  measures  will  be  used.  These  two  types  of  scores  will  be  analyzed  for  positive 
correlation.  The  data  analysis  from  survey  studies  will  help  us  to  determine  the 
soundness,  meaningfulness  and  applicability  of  the  software  project  management 
effectiveness  metric  (PME).  The  framework  of  the  research  is  shown  in  Figure  1. 
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Figure  1.  Research  Framework 
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3. 


Research  Scope 


The  scope  of  the  research  includes: 

•  A  background  analysis  and  literature  review. 

•  Development  of  a  viable  framework  for  software  project  management. 

•  Validation  of  the  framework  through  a  survey  study. 

•  Identification  of  significant  aspects  of  software  project  management. 

•  Development  of  a  metric  that  measures  the  project  management 
effectiveness  in  software  projects. 

•  Validation  of  the  metric  through  studies  on  software  projects. 

H.  ASSUMPTIONS  AND  APPLICABILITY 

It  is  important  to  state  the  assumptions  and  applicability  of  the  measurement 
activity  clearly.  Therefore,  assumptions  and  applicability  of  the  metric  are  provided 
below. 


1.  Assumptions 

Assumption  1.  A  software  project  development  requires  at  least  an  informal 
process  that  involves  certain  activities. 

Assumption  2.  Software  project  management  requires  certain  concepts,  entities, 
roles  and  functions  to  exist  within  the  software  project  team  and  the  rest  of  the 
stakeholders. 

Assumption  3.  Small  size  maintenance  projects  require  different  management 
techniques  than  software  development  projects.  Therefore,  the  metric  may  not  be  reliable 
for  small  software  maintenance  projects. 

Assumption  4.  There  exists  at  least  one  person  who  has  insight  over  a  broad  aspect 
of  project  management  and  the  metric  instrument  is  to  be  used  by  such  a  person. 
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2.  Applicability 


Applicability  1.  The  metric  instrument  is  designed  to  be  applicable  to  software  or 
software  intensive  projects.  It  is  not  applicable  to  projects  in  which  only  a  very  small 
portion  of  the  project  involves  software  development. 

Applicability  2.  The  metric  is  applicable  to  canceled  projects  on  the  condition  that 
at  least  some  requirements  development  activities  are  conducted. 

Applicability  3.  The  instrument  measures  the  project  management  effectiveness 
from  the  project  start  to  the  customer  delivery  time  or  the  time  it  is  canceled.  The  project 
start  time  is  defined  (for  the  purpose  of  this  metric)  as  the  time  after  the  business  decision 
is  made  to  go  ahead  with  the  project.  Therefore,  the  soundness  and  quality  of  the  business 
decision  is  not  included  in  this  measurement.  Because  the  metric  instrument  evaluates  the 
management  effectiveness  during  the  development,  the  business  decision  is  not 
considered  a  part  of  the  development  for  the  purpose  of  this  study.  The  quality 
assessment  of  the  business  decision  may  require  different  metrics. 

The  measure  of  project  management  effectiveness  may  be  high  even  though  the 
customer  never  uses  the  deliverables  of  the  project  due  to  various  reasons.  The 
assessment  of  customer  satisfaction  after  delivery  is  not  included  in  the  measurement. 

Applicability  4.  The  metric  is  not  designed  to  evaluate  the  effectiveness  of  small 
software  maintenance  projects. 

Applicability  5.  The  metric  is  not  applicable  to  very  small  software  development 
efforts  in  which  certain  management  roles  and  functions  are  not  distinctively  identifiable. 
These  projects  generally  include  a  very  small  team  of  developers. 

Applicability  6.  The  metric  instrument  should  be  used  by  a  project  manager,  an 
executive  manager  or  a  project  team  member  who  has  broad  insight  into  all  management 
aspects  of  the  project. 
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I.  KEY  DEFINITIONS:  PROJECT,  SOFTWARE  PROJECT,  PROJECT 
MANAGEMENT,  AND  SOFTWARE  PROJECT  MANAGEMENT 

First  of  all,  we  have  to  define  what  a  project  is  and  discuss  if  the  definition  applies 
to  software  projects.  There  are  various  definitions  of  projects  in  the  literature.  One  of  the 
best  is  offered  by  Tuman  (1988): 

A  project  is  an  organization  of  people  dedicated  to  a  specific  purpose  or 
objective.  Projects  generally  involve  large,  expensive,  unique,  or  high  risk 
undertakings  which  have  to  be  completed  by  a  certain  date,  for  a  certain 
amount  of  money,  within  some  expected  level  of  perfonnance.  At  a 
minimum,  all  projects  need  to  have  well  defined  objectives  and  sufficient 
resources  to  carry  out  all  the  required  tasks. 

Some  of  the  recognized  researches  on  various  aspects  of  projects  are  reported  by 
Jeffrey  K.  Pinto  and  Dennis  P.  Slevin  dating  back  to  the  1980s.  Pinto  and  Slevin  (1988) 
defined  the  characteristics  of  projects  as  follows: 

A  defined  beginning  and  end  (specified  time  to  completion). 

A  specific,  preordained  goal  or  set  of  goals  (performance  expectations). 

A  series  of  complex  or  interrelated  activities. 

A  limited  budget. 

All  of  these  project  characteristics  also  apply  to  software  projects  without 
exception.  An  addition  to  these  characteristic  is  as  follows: 

The  project  emphasis  must  be  on  software  or  a  mix  of  software  and  hardware. 

The  definition  of  a  software  project  for  the  purpose  of  this  dissertation  is  as 
follows: 

A  software  project  is  an  undertaking  in  which  the  emphasis  is  a  piece  of 
software  or  a  mix  of  software  and  hardware  and  it  involves  a  series  of 
complex  or  interrelated  activities  to  achieve  a  specific,  preordained  goal 
or  set  of  goals.  The  software  project  has  a  limited  budget,  a  defined 
beginning  and  an  end. 

Munns  and  Bjeirmi  (1996)  state  that  the  distinction  between  the  project  and 
project  management  is  less  than  precise.  Definitions  for  these  two  terms  are  also  provided 
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and  their  definition  of  a  project  is  quite  similar  with  the  given  definition  above.  The 
definition  of  project  management  is  given  by  Munns  and  Bjerimi  (1996)  is: 

...project  management  can  be  defined  as  the  process  of  controlling  the 
achievement  of  the  project  objectives.  Utilizing  the  existing  organizational 
structures  and  resources,  it  seeks  to  manage  the  project  by  applying  a 
collection  of  tools  and  techniques,  without  adversely  disturbing  the  routine 
operation  of  the  company. 

The  functions  of  project  management  include: 

Defining  what  needs  to  be  done  in  order  to  achieve  project  goals. 

Establishing  the  boundaries  and  extent  of  work. 

Determining,  planning,  estimating  and  allocating  the  resources  required. 

Planning  and  implementing  the  work. 

Monitoring  the  progress  of  work. 

Managing  risk,  adjusting  and  accommodating  deviations  from  the  plan. 

The  definitions  of  project  and  project  management  may  seem  to  overlap  in  many 
aspects.  Both  are  oriented  towards  the  accomplishment  of  the  project.  However,  Munns 
and  Bjeirmi  (1996)  point  out  an  important  difference  between  these  two.  While  the  scope 
of  a  project  is  long-term,  the  scope  of  project  management  is  short-term.  The  expected 
benefits  from  a  project  may  be  financial,  technical  or  marketing-oriented.  All  of  these 
benefits  tend  to  be  long-tenn  in  nature.  For  example,  in  order  to  detennine  the  return  on 
investment  from  the  project,  a  certain  point  of  time  must  be  reached  after  the  project  is 
successfully  in  place.  If  the  return  on  investment  is  computed  just  after  the  project 
implementation,  it  is  unlikely  to  reflect  the  correct  figure.  Also,  the  project  success 
factors  and  the  perception  of  project  success  change  over  time  (Pinto  and  Slevin,  1988; 
Pinto  and  Prescott,  1988;  Pinto  and  Mantel,  1990;  Fortune  and  White,  2006).  On  the 
other  hand,  project  management  is  oriented  towards  planning  and  control.  The  basic 
concerns  are  on-time  delivery,  expenditures  within  budget  expectations,  and  achieving 
the  necessary  expected  performance.  All  these  are  short-tenn  goals  in  the  life  cycle  of  a 
project.  After  the  project  delivery  for  use,  project  management  tasks  are  either  completed 
or  significantly  reduced.  Thus,  project  management  success  should  be  measured  upon 


15 


project  delivery  when  most  project  management  tasks  are  completed.  Software  project 
management  refers  to  the  project  management  when  the  project  emphasis  is  on  software 
or  a  mix  of  software  and  hardware. 
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II.  RELATED  WORK 


The  research  on  success  and  effectiveness  of  project  management  has  been  the 
interest  of  various  researchers  in  the  past.  However,  most  of  the  research  completed  was 
not  specifically  targeted  to  the  software  development  field.  The  projects  analyzed  in  these 
studies  included  projects  from  diverse  fields  such  as  construction,  manufacturing, 
environmental,  etc.  It  has  been  recognized  that  managing  software  projects  is  different  in 
many  aspects  (Brooks,  1995).  However,  it  is  possible  to  extract  similarities  that  can  help 
us  to  conduct  studies  on  software  project  management  effectiveness. 

A.  DISCUSSION  OF  PROJECT  SUCCESS 

First,  it  is  important  to  emphasize  that  project  success  is  not  the  same  as  project 
management  success  (Cooke-Davies,  2004c). 

Defining  project  success  is  not  an  easy  task.  According  to  Griffin  and  Page 
(1996),  it  is  multifaceted  and  difficult  to  measure.  Even  though  many  studies  have  been 
conducted  on  identification  of  project  success  factors,  project  failure  factors  or  related 
areas,  the  criteria  for  success  have  not  been  well-established.  Pinto  and  Slevin  (1998,  p. 
379)  state  that  “success”  and  “failure,”  like  beauty,  are  in  the  eyes  of  the  beholder.  There 
is  a  significant  risk  of  mislabeling  projects  as  success  or  failure  when  there  are  no 
universally  agreed  criteria.  Pinto  and  Slevin  argued  the  need  for  a  working  definition  of 
project  success.  The  three  conventional  criteria  of  project  success  present  challenges. 
These  conventional  criteria  are: 

•  Time  (completing  the  project  within  the  scheduled  time  frame). 

•  Cost  (completing  the  project  within  its  budget  limits). 

•  Performance  (completing  the  project  with  achieving  its  intended  mission 
and  specifications). 
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Pinto  and  Rouhiainen  (1998)  state  that  these  criteria  do  not  work  in  the  modern 
business  world.  The  tremendous  competition  in  this  modern  business  world  requires  a 
customer-oriented  focus.  Therefore,  customer  satisfaction  should  be  a  criterion  in  the 
evaluation  of  project  success. 

Glass  (1999)  points  out  the  need  for  a  new  theory  of  project  success.  Different 
stakeholders  may  have  different  concerns.  This  is  inevitable.  One  of  the  key  challenges  of 
any  project  management  is  to  align  the  goals  and  address  the  concerns  of  the 
stakeholders.  Linberg  (1999)  showed  that  the  definition  of  success  for  software 
practitioners  is  quite  different  from  the  conventional  criteria.  Software  practitioners  may 
classify  a  project  as  a  success  even  though  it  is  late,  or  perhaps  over  budget.  They  are 
more  concerned  with  the  quality  and  functionality  of  the  product.  Also,  they  may  even 
view  a  cancelled  project  as  a  success  due  to  the  lessons  learned  from  the  project.  Agarwal 
and  Rathod  (2006)  investigated  the  notion  of  software  project  success  for  different 
stakeholders.  They  examined  project  success  in  the  views  of  programmers/developers, 
project  managers,  and  customer  account  managers.  Procaccino  (2002,  et  ah,  2005) 
developed  a  quantitative  model  for  early  assessment  of  software  development  success  in 
the  practitioner’s  perspective. 

Griffin  and  Page  (1996)  suggest  that  the  most  appropriate  set  of  success  measures 
should  be  derived  from  the  project  strategy.  For  example,  the  success  criteria  for  a 
product  development  that  opens  up  a  new  market  should  be  different  than  the  criteria  for 
the  development  of  a  product  that  is  extending  a  product  line.  They  relate  the  product 
development  success  to  the  company’s  innovation  strategy. 

Cooke-Davies  (2004a)  examines  the  issue  with  a  broader  view.  His  view 
beautifully  clarifies  some  challenged  research  areas.  He  provides  a  definition  of  success 
at  different  levels.  His  questions  for  each  level  help  us  to  focus  on  the  heart  of  the  matter. 
According  to  him,  there  are  three  levels  of  success: 

•  Level  1 .  Project  Management  Success  -  was  the  project  done  right? 

At  this  level,  the  measures  of  success  are  the  traditional  project  success 

criteria.  The  job  of  project  management  is  about  managing  time,  cost  and 

quality.  The  project  management’s  job  should  start  after  the  decision  about 
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the  business  case  has  been  concluded  by  upper  management.  Therefore, 
the  question  “was  the  project  done  right?”  really  suggests  what  the 
measure  of  success  at  this  level  is.  The  focus  of  this  study  is  the  first  level 
of  success  which  is  project  management  success. 

•  Level  2.  Project  Success  -  was  the  right  project  done? 

This  level  of  success  is  about  choosing  the  right  problem  to  solve. 

•  Level  3.  Consistent  Project  Success  -  were  the  right  projects  done  right, 
time  after  time? 

This  level  of  success  is  about  organizational  success.  In  order  to  reach 
organizational  success,  organizations  have  to  complete  projects 
successfully  over  and  over. 

B.  PROJECT  SUCCESS  FACTORS 

This  dissertation  focuses  on  software  project  management  effectiveness, 
evaluation,  and  the  development  of  a  metric.  The  literature  on  the  subject  is  quite  limited. 
However,  it  is  possible  to  draw  similarities  from  researches  on  project  management  and 
project  success.  These  studies  mostly  focus  on  the  broad  aspects  of  projects.  A  brief 
overview  of  selected  studies  on  project  success  will  be  provided  next. 

A  significant  part  of  literature  on  project  management  is  the  identification  of 
critical  factors  for  project  success  or  successful  project  implementation.  Most  of  them  are 
theoretically  based  and  only  some  fraction  of  them  is  empirically-supported.  One  of  the 
earliest  attempts  was  conducted  by  Pinto  and  Slevin  (1987).  They  identified  critical 
success  factors  felt  to  be  predictive  of  successful  project  management.  Ten  factors  were 
discovered  after  an  empirical  study.  Some  of  these  factors  are  also  mentioned  in  other 
theoretical  research  (Martin,  1976;  Locke,  1984;  Cleland  and  King,  1983;  Sayles  and 
Chandler,  1971;  Baker,  Murphy  &  Fisher,  1983;  DeCotiis  and  Dyer,  1979).  The  ten 
factors  are: 
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Project  Mission:  Having  clearly  defined  goals  was  rated  as  one  of  the  most 
important  factors.  This  factor  is  supported  with  many  research  works  (Martin,  1976; 
Locke,  1984;  Cleland  and  King,  1983;  Baker,  Murphy  &  Fisher,  1983;  Pinto  and 
Kharbanda,  1995). 

Top  Management  Support:  This  factor  is  considered  as  a  key  enabler  in  many 
studies  for  successful  project  implementation.  It  is  considered  of  great  importance  in 
determining  the  ultimate  success  or  failure  of  projects  (Schultz  and  Slevin,  1975).  Top 
management  support  includes  aspects  such  as  allocation  of  sufficient  resources  including 
financial,  manpower,  time,  etc.  The  degree  of  management  support  will  lead  to 
considerable  variations  in  the  acceptance  of  the  project  by  stakeholders  (Manley,  1975). 
Two  important  studies  of  The  Standish  Group  (1994,  2000)  discovered  that  top 
management  support  in  IT  projects  is  among  the  first  two  success  factors. 

Project  Schedule/Plan:  This  factor  refers  to  all  planning  and  scheduling  activities 
including  contingency  plans  in  case  the  project  is  off  schedule.  It  also  includes  risk 
management  issues  related  to  budget  and  manpower.  This  factor  may  be  categorized 
under  different  names  or  divided  into  some  other  factors  in  other  studies. 

Client  Consultation:  In  Pinto  and  Slevin  (1987),  the  client  refers  to  anyone  who 
will  ultimately  be  making  use  of  the  project  as  either  a  customer  outside  the  company  or  a 
department  within  the  organization.  Manley  (1975)  found  that  client  involvement  to  the 
project  creates  significant  variations  in  their  support  to  the  project.  Like  the  project 
schedule/plan  factor,  client  consultation  can  be  found  in  many  research  works  under 
different  names.  For  software  engineering  research,  the  meaning  is  close  to  consideration 
of  the  stakeholders’  interests.  Under  The  Standish  Group  (1994,  2000),  the  factor 
partially  exists  as  user  involvement. 

Personnel  Issues:  The  fifth  factor  was  personnel  issues  including  recruitment, 
selection,  and  training.  Some  research  suggests  that  the  right  people  for  the  right  job  is  an 
enabler  for  successful  project  implementation  (O’Connell,  2002).  For  example, 
Hammond  (1979)  included  people  as  a  variable  to  his  contingency  model  of  the 
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implementation  process.  Also,  the  importance  of  project  manager  skills  is  emphasized  in 
some  studies  (Sayles  and  Chandler,  1971;  Baker,  Murphy  &  Fisher,  1983;  Belasi  and 
Tukul,  1996;  Vemer  and  Evanco,  2005). 

Technical  Tasks:  This  factor  refers  to  the  necessity  of  having  personnel  with  the 
necessary  technical  skills  and  having  the  adequate  technology  to  perfonn  their  tasks. 

Client  Acceptance:  Client  acceptance  refers  to  the  final  stage  of  the 
implementation  process.  Even  though  client  consultation  is  managed  well  during  the 
project  development,  client  acceptance  is  another  step  to  be  managed  just  like  other 
stages.  A  study  was  conducted  by  Bean  and  Radnor  (1979)  to  examine  intennediaries 
between  the  parties  of  the  project. 

Monitoring  and  Feedback:  This  is  the  eighth  factor  Pinto  and  Slevin  identified. 
Monitoring  and  feedback  refers  to  overseeing  the  schedule,  budget  and  perfonnance,  and 
taking  corrective  action  when  plans  are  deviating.  Souder  (1975)  emphasizes  the 
importance  of  constant  monitoring  from  a  budgeting  perspective.  With  metrics, 
monitoring  and  feedback  will  be  based  on  facts.  Putnam  and  Myers  (2003)  suggest  the 
use  of  metrics  to  manage  projects.  Reel  (1999)  also  emphasizes  that  keeping  track  of 
progress  during  software  development  and  post-mortem  analysis  are  crucial  for  success. 

Communication:  Having  proper  and  adequate  communication  channels  in  place 
between  stakeholders  is  extremely  essential  for  successful  project  implementation.  This 
factor  works  as  a  catalyst  for  many  other  factors  such  as  advertisement  of  project 
mission,  top  management  support,  client  consultation,  personnel  issues,  client  acceptance, 
etc. 

Troubleshooting:  The  last  factor  listed  in  the  Pinto  and  Slevin  (1987)  study  is 
trouble-shooting.  The  tenn  actually  refers  to  risk  management  activities  in  today’s 
literature.  The  importance  of  risk  management  is  broadly  recognized  today. 

The  study  also  recognizes  the  necessity  of  an  empirically  based  model  of  the 
project  implementation  process  and  a  measurement  instrument  to  quantify  the  success  of 
a  project  implementation.  Such  a  model  can  be  formulated  as  follows  (Pinto  and  Slevin, 
1987): 
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S  =  f(xi,  X2,  ....  ,  Xn ) , 

where  S  is  project  success,  and  xi  is  the  critical  success  factor 

In  this  simplified  model,  it  is  assumed  that  independent  variable  success  factor  xi 
relates  positively  to  project  success.  The  study  only  identifies  the  critical  success  factors; 
however,  it  doesn’t  measure  the  strength  of  their  relationship  with  project  success. 

Pinto  and  Slevin  conducted  empirical  research  on  project  implementation.  They 
identified  the  critical  success  factors  and  even  proposed  a  simple  measurement  model 
based  on  critical  factors.  The  importance  of  this  research  is  that  the  factors  identified  are 
more  comprehensive  than  most  other  studies  (Martin,  1976;  Locke,  1984;  Sayles  and 
Chandler,  1971;  Morris  and  Hough,  1987;  Reel,  1999).  This  study  partially  relates  to  this 
research  by  proposing  a  measurement  model.  Different  types  of  projects  require  different 
project  management  practices  (Cooke-Davies,  2004b).  Thus,  different  factors  may  be 
descriptive  of  project  success  in  different  projects.  In  addition,  the  focus  in  this 
dissertation  is  on  project  management  success,  which  can  be  considered  as  a  factor  in 
project  success  (Munns  and  Bjeinni,  1996;  Wohlin  and  Mayrhauser,  2001). 

Jiang  et  al.  (1996)  produced  a  list  of  thirteen  critical  success  factors  in  1996.  The 
factors  identified  were  almost  identical  to  those  identified  by  Pinto  and  Slevin. 
Competent  project  manger,  competent  project  team  members,  and  sufficient  resource 
allocation  may  be  thought  as  the  addition  to  the  previous  list.  The  study  ends  with  an 
important  conclusion  stating  that  information  system  users  and  professionals  are 
surprisingly  similar  in  their  importance  rankings  of  success  factors.  It  is  also  important  to 
note  that  the  success  factors  haven’t  changed  dramatically  over  time. 

Gemuenden  and  Lecher  (1997)  conducted  an  empirical  study  on  identifying 
critical  success  factors  based  on  a  large  data  set  (448  projects).  Their  goal  was  to  identify 
a  limited  number  of  factors.  They  have  identified  eight  success  factors  explaining 
approximately  59%  of  variance  in  project  success.  These  factors  are  top  management, 
project  leader,  project  team,  participation,  information/communication,  planning/control, 
conflicts,  and  goal  changes.  They  rely  on  the  participants’  view  while  determining 
whether  a  project  is  successful  or  not. 
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The  critical  success  factors  in  software  projects  are  reported  by  Reel  (1999). 
However,  the  factors  are  rather  experience-oriented  as  opposed  to  being  the  result  of  an 
empirical  study.  His  list  was  comprised  of  five  essential  factors  to  managing  a  successful 
project: 

Start  on  the  right  foot:  Field  (1997)  provided  ten  signs  of  information  systems 
project  failures,  of  which  at  least  seven  are  determined  at  the  start  of  the  project,  such  as 
project  managers  don’t  understand  users’  needs,  the  project’s  scope  is  ill-defined,  etc.  By 
resolving  such  issues,  we  can  improve  the  success  chance  of  the  projects.  Building  the 
right  team,  ensuring  that  there  are  enough  resources  for  the  project,  providing  the  highly 
productive  environment  and  the  necessary  tools,  and  involving  users  and  customers  are 
all  part  of  starting  on  the  right  foot.  Most  of  these  issues  can  be  found  under  different 
factors  in  Pinto  and  Slevin  (1987). 

Maintain  momentum:  Starting  on  the  right  foot  provides  momentum  for  the 
project  team.  However,  it  is  crucial  to  maintain  the  momentum  for  the  duration  of  the 
project.  There  are  three  issues  that  need  attention  under  this  title:  attrition,  quality  and 
management.  Reel  (1999)  points  out  the  attrition  problem  in  the  software  industry  and 
states  that  it  can  be  disastrous  for  a  mid-stream  software  project.  Brooks’  famous  law, 
“adding  manpower  to  a  late  software  project  makes  it  later,”  explains  why  it  can  be  a 
disaster.  Quality  must  be  incorporated  throughout  the  development  process.  It  is  not 
possible  to  go  back  and  add  quality.  Reel  (1999)  recommends  managing  the  product  more 
than  the  personnel.  The  observation  he  had  was  that  project  leaders  often  avoid 
confronting  individuals  and  merely  fix  a  problem  by  setting  arbitrary  rules. 

Track  progress:  There  is  an  important  difference  between  civil  engineering 
projects  and  software  projects.  The  progress  of  a  construction  project  can  easily  be 
observed;  however,  due  to  the  intangible  aspect  of  software  projects,  it  is  hard  to  observe 
the  status  of  a  software  project.  But  this  doesn’t  eliminate  the  necessity  of  tracking 
progress  during  software  development.  There  are  methods  to  accomplish  it,  and  we  have 
to  get  the  most  out  of  them. 
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Make  smart  decisions:  Some  key  decisions  about  the  projects  determine  whether 
the  project  will  be  successful  or  not.  For  example,  designing  a  networking  protocol  and 
building  a  communication  tool  may  cost  much  more  than  buying  a  commercial-off-the- 
shelf  tool.  At  best,  buying  a  commercial-off-the-shelf  product  may  cost  a  fortune; 
however,  it  may  also  be  the  decision  that  saves  the  project.  Good  project  leaders  are  the 
ones  that  can  make  the  smart  choices. 

Institutionalize  post-mortem  analyses:  Without  figuring  out  what  happened  during 
a  project,  it  is  inevitable  to  repeat  the  same  mistakes  over  and  over  again.  The  difference 
between  CMMI  level- 1  and  other  CMMI  levels  are  institutionalization  (CMMI,  2002). 
CMMI  level- 1  is  named  as  Initial  which  is  a  rating  when  organizations  can  not  be  rated 
with  higher  levels.  CMMI  level-2  is  a  rating  given  to  organizations  when  the  existence  of 
certain  processes  and  practices  only  warrant  for  this  level.  CMMI  level  3,  4  and  5  focuses 
on  improvement  of  the  process,  and  improvement  can  only  be  achieved  by  collecting  and 
analyzing  project  data.  Process  improvement  movement  inspired  by  Dr.  Deming  validates 
the  benefit  of  measuring  a  process  and  improving  it  (Aguayo,  1990). 

The  factors  identified  in  this  study  overlap  some  of  the  factors  listed  earlier  (Pinto 
and  Slevin,  1987).  So,  it  is  imperative  to  say  that  there  are  commonalities  between 
success  of  any  other  types  of  projects  and  software  projects.  However,  when  software 
projects  and  other  types  of  projects  are  compared,  the  likelihood  of  having  an 
unsuccessful  result  in  a  software  project  is  quite  high.  This  suggests  that  there  are  some 
critical  factors  creating  this  huge  variation  in  the  success  of  software  projects.  The  clues 
Reel  (1999)  provides  us  are: 

•  Managing  complexity  of  a  software  development  project. 

•  Attrition  in  the  software  industry. 

•  Quick  technological  trend  shifts  and  making  smart  decisions. 

•  The  issue  of  adding  overall  quality  to  software. 

Reel’s  report  relates  to  this  dissertation  by  identifying  factors  for  managing 
software  projects.  Nonetheless,  this  report  was  not  supported  by  an  empirical  study.  In 
addition,  he  does  not  bring  up  the  issue  of  effectiveness  measurement. 
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C.  COMPREHENSIVE  LIST  AND  COMPARISON  OF  PROJECT  SUCCESS 
FACTORS 

Fortune  and  White  (2006)  conducted  a  recent  study  which  includes  a 
comprehensive  literature  search  on  critical  project  success  factors.  The  goal  of  the  study 
was  to  frame  project  critical  success  factors  by  a  systems  model.  In  the  study,  they 
reviewed  63  publications  and  extracted  26  different  factors.  The  factors  found  are  listed 
in  decreasing  order  of  frequency  of  occurrence  (the  numbers  in  parenthesis  are  the  counts 
of  occurrence  in  different  publications)  (Fortune  and  White,  2006): 

•  Support  from  senior  management  (39) 

•  Clear  realistic  objectives  (31) 

•  Strong/detailed  plan  kept  up  to  date  (29) 

•  Good  communication/feedback  (27) 

•  User/client  involvement  (24) 

•  Skilled/suitably  qualified/sufficient  staff/team  (20) 

•  Effective  change  management  (19) 

•  Competent  proj  ect  manager  (19) 

•  Strong  business  case/sound  basis  for  project  (16) 

•  Sufficient/well  allocated  resources  (16) 

•  Good  leadership  ( 1 5) 

•  Proven/ familiar  technology  (14) 

•  Realistic  schedule  (14) 

•  Risks  addressed/assessed/managed  (13) 

•  Project  sponsor/champion  (12) 

•  Effective  monitoring/control  (12) 

•  Adequate  budget  (11) 

•  Organizational  adaptation/culture/structure  (10) 

•  Good  performance  by  suppliers/contractors/consultants  ( 1 0) 

•  Planned  close  down/review/acceptance  of  possible  failure  (9) 

•  Training  provision  (7) 
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•  Political  stability  (6) 

•  Correct  choice/past  experience  of  project  management  (6) 

•  Environmental  influences  (6) 

•  Learning  from  past  experience  (5) 

•  Project  size  (large)/  level  of  complexity  (high)/  number  of  people  involved 
(too  many)/  duration  (over  3  years)  (4) 

•  Different  viewpoints  (3). 

The  three  most  cited  factors  are  found  in  more  than  80%  of  the  publications. 
However,  only  17%  of  the  publications  cite  all  three  of  them  (Fortune  and  White,  2006). 
There  is  a  lack  of  consensus  on  the  factors.  The  lack  of  consensus  on  the  factors 
influencing  project  success  was  also  identified  by  Wateridge  (1995).  Fortune  and  White 
criticize  the  critical  success  factors  approach.  One  of  the  two  criticisms  they  had  is  that 
the  inter-relationships  between  factors  are  also  as  important  as  the  factors  themselves, 
which  may  also  be  the  reason  on  the  variations  of  factors  identified  by  different 
researchers.  The  other  one  is  viewing  the  project  implementation  as  a  static  process. 
However,  project  implementation  is  a  dynamic  phenomenon  and  different  factors  have 
varying  importance  on  different  levels  and  stages.  Pinto  and  Mantel  (1990)  showed  that 
the  critical  factors  associated  with  success  and  failure  varies  during  the  project  life  cycle. 
While  mission,  top  management  support,  and  schedule/plans  have  significance  during  the 
strategic  stage,  factors  such  as  client  consultation,  personnel,  technical  tasks  and  others 
are  important  in  the  tactical  stage. 

Fortune  and  White  (2006)  propose  a  new  solution  for  analysis  and  predicting  the 
outcomes  of  the  projects.  The  solution  consists  of  a  system  model  approach.  The  model  is 
known  as  Fonnal  System  Model  (FSM)  and  it  is  developed  by  Bignell  and  Fortune 
(1984).  The  Formal  System  Model  consists  of  a  decision-making  subsystem,  a 
performance-monitoring  subsystem  and  subcomponents  and  components  that  carry  out 
transfonnations.  The  model  also  describes  the  interactions  between  subcomponents  and 
environment.  Fortune  and  White  (2006)  mapped  all  the  critical  success  factors  identified 
in  the  literature  to  concepts  in  the  Formal  System  Model.  Then,  the  model  is 
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experimented  on  two  projects.  Since  all  the  factors  are  mapped,  they  analyzed  the 
projects  with  the  FSM  and  showed  that  the  model  is  capable  of  predicting  the  outcomes 
of  the  projects. 

The  study  conducted  by  Fortune  and  White  (2006)  relates  to  this  research  by 
trying  to  predict  the  outcomes  of  projects  in  two  simple  measures:  success  or  failure.  This 
evaluation  can  also  be  considered  as  a  binary  measurement.  However,  this  measurement 
has  limited  capabilities  to  determine  how  successful  the  project  is. 

D.  PROJECT  FAILURE  FACTORS 

Another  line  of  research  is  about  failure  factors  in  projects.  One  well-known  study 
known  as  the  CHAOS  study  was  conducted  by  The  Standish  Group  (1994).  The  study 
includes  both  success  and  failure  factors.  The  failure  factors  were  divided  in  two  as 
project  challenged  factors  and  project  impaired  factors.  Project  challenged  factors  are  the 
ones  that  reduces  the  effectiveness  of  the  project.  Lack  of  user  input,  incomplete 
requirements  and  specifications,  changing  requirements  and  specifications  are  the  first 
three  factors  in  the  list.  Project  impaired  factors  are  the  ones  that  cause  cancellation  of 
projects.  The  list  includes  incomplete  requirements,  lack  of  user  involvement,  lack  of 
resources,  unrealistic  expectations,  lack  of  executive  support,  changing  requirements  and 
specifications,  lack  of  planning,  project  no  longer  needed,  lack  of  information  technology 
management  and  technology  illiteracy  (The  Standish  Group,  1994).  This  study  is  simply 
a  report  of  the  statistics  gathered  from  a  large  database  of  projects.  It  does  not  contain 
detailed  research  and  explanations  on  the  reasons  of  successes  and  failures  of  software 
projects.  Another  report,  generated  by  the  same  Standish  Group  in  2000,  is  similar  to  the 
first  report  and  contains  updated  statistics. 

At  first,  failure  factors  may  be  simply  thought  as  the  lack  of  success  factors. 
However,  this  common  belief  is  not  true.  In  the  literature,  there  are  differences  in  success 
and  failure  factors.  For  example,  Pinto  and  Mantel  (1990)  list  different  factors  for  success 
and  failure  at  different  stages  of  project  implementation. 
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E.  THE  ROLE  OF  PROJECT  MANAGEMENT  IN  ACHIEVING  PROJECT 

SUCCESS 

The  terms  project  and  project  management  are  generally  confused  with  each 
other.  The  goal  in  both  endeavors  is  project  success.  Even  though  these  two  terms 
significantly  overlap,  there  is  an  important  distinction.  Understanding  this  distinction  and 
the  role  of  project  management  in  achieving  project  success  will  help  us  to  focus  our 
research  efforts  in  the  right  direction.  Munns  and  Bjeinni  (1996)  explained  the  distinction 
and  the  overlap  in  these  two  terms. 

The  distinction  of  these  terms  lies  in  their  scope  in  the  life  cycle  of  a  project.  The 
goals  in  a  project  are  towards  long-tenn  benefits  such  as  return  on  investment, 
productivity  increase  due  to  the  use  of  project,  etc.  However,  project  management  is 
generally  concerned  about  short-tenn  goals  such  as  on-time  delivery,  meeting 
performance  standards,  project  development  within  budget  expectations,  etc.  After  the 
deployment  of  the  project,  the  project  management  functions  are  generally  no  longer 
needed  or  reduced  to  minimum  just  for  maintenance  of  the  project.  Having  this  clear 
distinction  about  the  scope  of  both  concepts  makes  it  possible  to  discuss  the  distinction 
between  project  success  and  project  management  success.  A  developed  model  of  project 
success  is  shown  below  in  Figure  2.  The  model  briefly  depicts  the  success  measures  from 
different  perspectives. 


Figure  2.  A  Model  of  Project  Success  [From  (Pinto  and  Slevin,  1988)] 


28 


Since  project  success  is  oriented  towards  long-term  goals,  important  parameters 
within  the  goals  will  be  return  on  investment,  profitability,  competition  and  marketability 
(Munns  and  Bjeirmi,  1996).  As  shown  earlier,  the  list  of  critical  success  factors  is  long. 
Different  researchers  identified  different  factors  to  be  of  importance.  There  are  two  main 
reasons  for  such  differences.  First,  success  is  about  perception.  Stakeholders  may  have 
different  perceptions  of  success.  For  example,  learning  experience  from  technical 
challenges  or  a  comfortable  working  environment  may  constitute  a  success  for 
practitioners  (Procaccino,  2002;  Procaccino  et  ah,  2005;  Glass,  1999).  Client  and 
development  teams  have  different  concerns  relating  to  success  (Pinto  and  Slevin,  1988). 
Only  some  of  the  success  parameters  are  in  control  of  the  project  development  team 
(Munns  and  Bjeirmi,  1996).  There  are  factors  related  to  the  external  environment  such  as 
political,  economical,  social  and  technological  environments,  competitors,  sub¬ 
contractors,  etc.,  (Belasi  and  Tukul,  1996).  All  these  factors  create  the  differences  in 
perception  from  some  stakeholder’s  perspective.  Which  parties  are  interested  in  which 
part  of  project  life  cycle  is  given  below  in  Figure  3. 
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Figure  3.  The  Stages  of  Project  Life  Cycle  and  Associated  Parties  to  Each  Stage  [From 

(Munns  and  Bjeinni,  1996)] 

Another  reason  for  the  variance  is  that  project  success  and  project  success  factors 
change  over  time  (Pinto  and  Slevin,  1988;  Pinto  and  Prescott,  1988).  Project  success 
should  be  thought  of  as  a  dynamic  entity  rather  than  a  static  entity  (Fortune  and  White, 
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2006).  Figure  4  shows  the  assessment  of  project  success  over  time.  During  early  stages  of 
the  projects,  internal  factors  influence  the  project  more  than  external  factors,  therefore 
they  are  more  important.  Also,  success  measurement  is  easier.  The  percentage  of 
completed  project  work  at  a  given  stage  can  be  a  measure  of  success  during  project 
development.  For  example,  after  installation,  if  profitability  is  the  measure  of  success,  it 
is  subject  to  an  economical  environment  which  may  fluctuate  over  time.  Therefore,  it  is 
important  to  know  when  to  measure  success  in  a  project  life  cycle  and  from  what 
perspective.  In  Figure  4,  the  top  line  shows  how  the  result  of  project  success  assessment 
changes  over  time  depending  on  the  factors  affecting  it.  Up  until  installation,  the  success 
assessment  function  is  linear  and  it  is  mostly  affected  by  internal  factors.  In  simple 
words,  the  success  may  be  measured  by  the  amount  of  work  accomplished.  After 
installation,  the  success  assessment  is  mostly  affected  by  external  factors.  The  assessment 
of  success  is  complex  and  can  fluctuate  rapidly,  especially  from  the  perspective  of  users 
and  customers.  After  installation,  new  bugs  in  software  applications  may  be  found  by  the 
system  users.  The  frustration  due  to  bugs  in  the  system  changes  the  perception  of  the 
user,  causing  the  user  to  perceive  the  product  as  a  failure.  Maintenance  and  bug  fixes  on 
the  system  replace  the  negative  perception  of  the  users  with  a  positive  perception  again. 
This  cycle  continues  until  the  end  of  the  life  cycle  of  the  application. 


Figure  4.  Assessment  of  Project  Success  over  Time  [After  (Pinto  and  Slevin,  1988)] 
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The  obvious  success  measures  for  project  management  are  completion  of  the 
project  in  time,  within  budget,  adequate  quality  standards,  and  meeting  the  project  goal. 
Project  management  success  is  a  part  of  project  success.  However,  it  is  possible  to 
achieve  success  in  a  project  even  though  project  management  fails,  or  vice  versa  (Wit, 
1988).  Thus,  we  have  to  differentiate  the  measurement  of  project  success  from  project 
management  success.  The  scopes  of  project  management  success  and  project  success  are 
provided  in  Figure  5.  In  order  to  conduct  a  successful  measurement,  we  have  to  measure 
concepts  within  their  scope. 
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Figure  5.  The  Scope  of  Success  within  Project  Life  Cycle  [From  (Munns  and  Bjeirmi, 

1996)] 

F.  MEASUREMENT  OF  PROJECT  SUCCESS 


One  of  the  early  attempts  to  define  and  measure  project  perfonnance  is  provided 
by  DeCotiies  and  Dyer  (1979).  The  study  was  aimed  to  conceptualize  and  measure  the 
dimensions  of  project  perfonnance  and  their  determinants.  It  was  conducted  in  a  high 
technology  matrix  organization  in  an  effort  to  increase  effectiveness  of  project  groups.  A 
questionnaire  was  developed  in  order  to  identify  the  dimensions  of  project  performance. 
The  study  identified  and  defined  five  distinct  performance  dimensions: 
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Manufacturability  and  business  performance:  Extent  to  which  the  product  is 
manufacturable,  and  finished  in  time  to  make  a  timely  market  entry  and  result  in  a 
favorable  financial  return. 

Technical  performance:  Extent  to  which  the  project  generates  the  needed 
technical  data  and  critical  technical  specifications  are  met. 

Efficiency:  Extent  to  which  the  project  operates  efficiently  in  terms  of  costs,  time, 
and  productivity. 

Personal  growth  experience:  Extent  to  which  the  project  provides  those  involved 
an  interesting,  challenging,  and  professionally  developing  experience. 

Technological  innovativeness:  Extent  to  which  the  project  results  in  significant 
technological  advances. 

DeCotiis  and  Dyer  (1979)  selected  manufacturability  and  business  perfonnance  as 
their  ultimate  project  success  criterion.  Technical  performance  and  efficiency  were 
selected  as  major  contributors  to  project  success  and  technological  innovativeness  as  a 
minor  contributor.  Professional  growth  experience  would  be  a  supplementary  success 
criterion  if  other  perfonnance  dimensions  are  rated  high.  Projects  are  measured  in  each  of 
these  dimensions. 

Every  project  performance  dimension  is  measured  with  project  perfonnance 
determinants.  These  detenninants  of  project  performance  include  management  support, 
inter-organizational  relations,  sponsor  relations,  transfer  management,  panning  and 
stability  of  specifications  and  designs,  clarity  of  project  leader  role,  project  member’s 
skills  and  cooperation,  communication  and  decision-making,  and  personal  utilization, 
planning  and  scheduling,  control  procedures,  and  leadership. 

The  study  included  the  results  of  a  stepwise  regression  model  to  identify  the  most 
important  detenninants  for  each  dimension.  For  example,  the  most  descriptive 
determinants  for  manufacturability  and  business  performance  are  transfer  management, 
planning  and  scheduling,  lack  of  inter-organizational  relations,  and  lack  of  clarity  of 
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project  leader’s  role.  Other  dimensions  may  be  investigated  with  a  different  set  of 
determinants.  The  study  ends  with  a  model  of  determinants  of  project  performance. 

DeCotiis  and  Dyer  (1979)  reached  an  important  conclusion  stating  “the 
identification  of  these  distinct  dimensions  of  project  perfonnance  illustrates  that  a  project 
can  be  both  successful  and  unsuccessful  at  the  same  time.”  For  example,  a  project  may  be 
innovative  from  a  technological  perspective;  however,  it  may  not  be  a  success  when  the 
perspective  is  manufacturability  and  business  performance. 

DeCotiis  and  Dyer’s  (1979)  study  is  important  in  terms  of  stating  the  complexity 
of  analyzing  project  success.  However,  the  model  developed  within  the  study  is  only  a 
broad  framework.  How  the  model  can  be  applied  to  different  types  of  projects  or  whether 
the  model  needs  modification  for  a  specific  type  of  project  are  some  of  the  questions  left 
outside  of  the  study.  DeCotiis  and  Dyer’s  study  relates  to  this  research  by  stating  the  fact 
that  measuring  project  performance  is  multidimensional.  While  Decotiis  and  Dyer’s 
study  is  focused  on  project  success,  this  dissertation  is  focused  on  project  management 
success.  Both  of  these  terms  have  several  overlapping  factors,  but  they  refer  to  different 
concepts.  In  addition,  in  this  research  the  emphasis  is  on  software,  unlike  in  Decotiis  and 
Dyer’s  study. 

Slevin  and  Pinto  (1986)  developed  and  tested  a  generalized  project  success 
measure  called  Project  Implementation  Profile  (PIP).  PIP  includes  a  questionnaire 
derived  from  critical  success  factors  identified  by  the  same  authors  (1987).  The 
questionnaire  is  applied  to  project  managers  and  uses  a  5-point  Likert  scale.  PIP  contains 
the  perspective  of  both  the  client  and  project  implementation.  The  measure  includes  two 
subscales  as  project  and  client  score  and  another  overall  score.  These  scores  can  then  be 
compared  according  to  a  score  ranking  of  completed  projects  database  of  418  projects.  If 
a  project  is  below  the  50th  percentile  in  any  factor,  the  project  manager  should  devote 
extra  attention  to  it.  A  table  of  the  percentile  scores  for  project  implementation  profile  is 
given  in  Appendix  A.  The  PIP  measurement  is  a  viable  method  to  measure  project 
success.  However,  its  scope  is  beyond  project  management,  which  is  the  emphasis  of  this 
dissertation. 
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After  releasing  their  well-known  report,  the  Standish  Group  (1995)  also 
developed  a  simple  measurement  technique  for  project  success.  The  Standish  Group  used 
the  ten  success  criteria  from  their  earlier  report  and  weighted  each  factor  based  on  a 
survey  of  IT  managers.  Careful  analysis  of  factors  will  reveal  that  all  factors  are  related  to 
project  management.  With  regard  to  the  discussion  presented  earlier  about  the  distinction 
between  project  and  project  management,  this  simple  measure  is  in  fact  a  software  project 
management  metric.  The  measurement  is  called  the  success  potential  chart.  There  is  a 
weight  associated  with  every  factor  in  the  success  potential  chart  with  the  total  weight  in 
the  chart  equaling  100.  Every  factor  is  divided  into  five  smaller  questions.  For  example, 
the  most  important  factor,  user  involvement,  consists  of  the  following  questions: 

Do  I  have  the  right  user(s)? 

Did  I  involve  the  user(s)  early  and  often? 

Do  I  have  a  quality  user(s)  relationship? 

Do  I  make  involvement  easy? 

Did  I  find  out  what  the  user(s)  needs  are? 

For  each  question  with  a  yes  answer,  3.8  points  should  be  added  to  the  success 
potential  score.  Other  factors  have  different  weights  and  the  same  procedure  is  followed 
throughout  the  assessment  to  calculate  the  success  potential  for  a  project.  The  chart  can 
be  found  in  Appendix  I. 

The  measurement  developed  by  the  Standish  Group  is  hardly  scientifically 
grounded.  Even  though  the  factors  are  a  result  of  an  empirical  study,  the  same  weight  for 
the  questions  within  the  factors  raises  validity  concerns.  However,  it  is  a  simple  measure 
and  it  can  still  differentiate  a  successful  project  management  from  a  failed  one. 

G.  MEASUREMENT  OF  SOFTWARE  PROJECT  MANAGEMENT 

EFFECTIVENESS 

As  pointed  out  earlier,  project  management  success  is  not  the  same  as  project 
success  (Cooke-Davies,  2004c). 

O’Connell  has  published  a  series  of  books  on  how  to  run  successful  projects.  In 
his  latest  book  (O’Connell,  2002),  he  presents  his  experiences  on  a  method  called 
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Structured  Project  Management.  The  method  relies  on  a  ten-step  progress  approach 
embedded  with  a  measurement  of  project  success.  The  measure  is  called  Project  Success 
Indicator  (PSI).  The  ten  steps  are  (O’Connell,  2002): 

1 .  Visualize  the  goal;  set  your  eyes  on  the  prize; 

2.  Make  a  list  of  things  to  be  done; 

3.  There  must  be  one  leader; 

4.  Assign  people  to  jobs; 

5.  Manage  expectations,  allow  a  margin  for  error,  have  a  fallback  position; 

6.  Use  an  appropriate  leadership  style; 

7.  Know  what’s  going  on; 

8.  Tell  people  what’s  going  on; 

9.  Repeat  Steps  1-8  until  step  10;  and  finally 

10.  The  prize. 

In  every  step,  there  are  sub-steps  helping  management  to  accomplish  the  project. 
There  are  scores  assigned  to  every  step  in  the  measure.  For  example,  the  first  two  steps 
are  assigned  20  out  of  100.  The  last  two  steps  correspond  to  a  score  of  0.  The  highest 
score  is  100.  As  a  general  guideline,  the  project  success  indicator  (PSI)  should  not  be 
below  60,  and  the  first  two  steps  are  the  most  important  ones.  O’Connell  (2002)  also 
explains  how  the  scoring  should  be  in  his  book. 

The  structured  project  management  approach  seems  to  be  a  viable  solution  for 
many  software  projects.  PSI  is  also  designed  to  assist  the  approach.  PSI  can  be 
considered  as  a  software  project  management  metric.  However,  PSI  is  purely  based  on 
the  general  guidelines  on  the  structured  project  management  and  mostly  subjective  on  the 
scoring.  The  information  PSI  provides  is  limited  in  many  tenns.  For  example,  when  the 
project  has  only  one  leader,  the  project  gets  10  points  on  the  scoring.  However,  the 
scoring  has  no  means  to  measure  the  effectiveness  of  the  leader  and  the  leadership. 
O’Connell  does  not  explicitly  claim  that  PSI  is  a  metric.  In  addition,  PSI  does  not  have 
the  ability  to  assess  the  project  management  success  with  precision.  On  the  other  hand, 
PSI  is  proactive  and  can  be  used  as  an  indicator  during  development  to  determine 
whether  the  project  management  is  becoming  a  success  or  not. 
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Osmundson  et  al.  (2003)  introduced  a  metric  called  the  quality  management 
metric  (QMM).  The  metric  considers  four  important  areas  of  project  development: 
requirements  management,  estimation/planning  management,  people  management,  and 
risk  management.  These  areas  within  software  projects  are  investigated  by  conducting 
surveys  on  project  managers  and  developers.  The  survey  reveals  a  quantitative  analysis  of 
the  project  which  can  be  used  for  improving  further  software  developments.  The  QMM 
metric  is  developed  by  Machniak  (1999).  Three  programs  are  used  to  validate  the 
proposed  metric.  The  validation  is  extended  by  Grossman  (2000)  by  applying  the  metric 
to  ten  more  Department  of  Defense  software  projects. 

These  initial  studies  showed  potential  that  a  project  management  metric  can  be 
developed.  The  studies  compared  the  QMM  metric  and  the  observed  program  success 
evaluated  by  the  program  managers  and  developers.  Such  comparison  was  the  base  for 
the  validation  of  the  QMM.  The  results  indicated  that  QMM  showed  a  strong  positive 
correlation  with  the  QMM  percentage  score  and  the  overall  program  score.  There  are 
mainly  two  issues  with  these  successful  studies.  First,  the  number  of  samples  in  these 
studies  is  limited.  Overall,  thirteen  software  programs  are  analyzed  in  these  studies.  More 
programs  are  needed  to  be  analyzed  to  fully  understand  the  applicability  of  this  metric. 

Within  the  software  programs  investigated  by  Machniak  and  Grossman,  there  are 
only  two  programs  involving  twenty-four  or  twenty-five  developers.  All  the  other 
programs  have  smaller  team  size.  This  data  set  calls  for  additional  research  to  reveal  the 
scalability  and  applicability  of  QMM  in  large-scale  projects. 

In  addition,  the  software  development  projects  are  all  defense-related  projects 
developed  by  military  research  centers.  The  expectation  is  that  these  centers  generally 
follow  specific  guidelines  set  forth  by  the  Department  of  Defense.  Another  concern  is 
QMM’s  applicability  or  assessment  success  in  a  commercial  environment  in  which 
practices  are  expected  to  differ  at  least  in  some  ways. 

H.  SUMMARY 

In  this  section,  related  work  was  presented.  The  related  work  presented  here  is  a 

collection  of  literature  from  a  variety  of  disciplines.  Some  of  these  disciplines  are 
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software  project  management,  project  management,  organizational  management, 
organizational  behavior,  and  organizational  sociology.  Defining  project  success, 
identification  of  project  success  and  failure  factors,  measurement  of  project  success,  and 
measurement  of  project  management  quality  studies  are  all  related  to  this  research. 

The  most  notable  research  related  to  this  study  is  the  development  of  a  quality 
management  metric.  The  metric  proposed  in  this  research  and  QMM  both  intend  to 
measure  the  same  concept,  which  is  software  project  management  quality  or 
effectiveness.  QMM  achieved  a  remarkable  success  in  capturing  the  essentials  of  project 
management  and  measuring  its  quality. 
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III.  APPROACHES  FOR  MEASURING  PROJECT 
MANAGEMENT  EFFECTIVENESS  IN  SOFTWARE  PROJECTS 

A.  INTRODUCTION 

There  are  various  studies  reporting  the  success  and  failure  rates  of  software 
projects  (GAO,  1979;  The  Standish  Group,  1995;  El  Emam  and  Koru,  2008).  Even  with 
the  lowest  failure  rates  reported,  software  projects  are  significantly  failing  when 
compared  to  projects  in  other  fields.  In  (Slevin,  Cleland  and  Pinto,  2002),  current  project 
management  issues  in  leading  project-based  industries  are  listed.  Among  nine  industries, 
only  in  the  software  industry  column  overruns  and  poor  perfonnance  are  explicitly  listed 
as  issues,  among  others.  The  average  software  project  is  likely  to  be  six  to  twelve  months 
behind  schedule  and  50  to  100  percent  over  budget  (Yourdon,  2004).  One  would  expect 
that  the  record  in  software  projects  should  have  been  much  better  with  all  the 
advancements  in  technical  aspects  of  software  engineering.  However,  relying  merely  on 
technological  advances  to  achieve  better  outcomes  in  software  projects  may  be 
misleading.  Significant  advances  in  software  project  management  field  to  achieve  better 
results  in  software  projects  are  also  required.  Therefore,  proposals  and  discussions  for 
applicable  and  viable  theories,  models,  tools  and  practices  in  software  project 
management  are  important  steps  in  achieving  better  project  outcomes. 

Ineffective  software  project  management  is  among  the  main  reasons  for  the 
failures  in  software  projects  (Jones,  2004).  In  addition,  effective  project  management  is  a 
determinant  in  the  success  of  the  software  projects  (Jones,  2004).  DeMarco  and  Lister 
(1999)  state,  “for  overwhelming  majority  of  the  bankrupt  projects  we  studied,  there  was 
not  a  single  technological  issue  to  explain  the  failure.”  Robertson  and  Robertson  (2005) 
emphasize  that,  “in  several  decades  of  project  experience,  we  have  never  seen  a  project 
fail  for  technical  reasons.  It  has  always  been  human  failures  that  have  caused  otherwise 
good  projects  grind  to  a  halt.”  Various  other  studies,  researchers  and  practitioners  report 
similar  issues  regarding  the  importance  of  software  project  management  in  the  success 
and  failure  of  software  projects  (Weinberg,  1994;  Defense  Science  Board,  2000). 
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According  to  Boehm,  poor  management  can  increase  software  costs  more  rapidly 
than  any  other  factor.  COCOMO,  a  method  for  software  project  cost  and  effort  estimation 
developed  by  Barry  Boehm  and  his  colleagues,  does  not  include  project  management  as  a 
factor  (Boehm,  1981).  Therefore,  in  COCOMO  II,  the  estimation  model  incorporates 
some  project  management  related  factors  such  as  PCON  (personnel  continuity)  and 
PMAT  (process  maturity)  (Boehm  et  ah,  2000).  We  believe,  in  order  to  keep  the  rate  of 
the  software  cost  overruns  and  schedule  slippages  down,  that  measuring  and  therefore 
improving  the  quality  of  project  management  areas  is  an  enabler.  In  addition,  such  project 
management  metrics  can  be  incorporated  to  cost  estimation  techniques  yielding  better 
estimates. 

According  to  Morris  (1998),  “one  of  the  major  areas  of  project  management 
development  over  the  next  years,  I  believe,  will  be  establishing  and  refining  inter¬ 
industry  metrics  for  quantifying  performance  improvements.  Much  of  this  work  will  be 
IT-related.”  Hyvari  (2006)  investigates  the  effectiveness  of  project  management  based  on 
four  different  factors.  The  factors  are  organizational  structures,  technical  competency, 
leadership  ability,  and  the  characteristics  of  an  effective  project  manager.  He  does  not 
state  the  reasoning  for  the  selection  of  these  factors  and  whether  this  is  a  complete  list  or 
not. 

Project  management  is  a  complex  endeavor  and  the  development  of  a  metric  for 
project  management  effectiveness  is  clearly  not  an  easy  task.  However,  measurement  and 
evaluation  of  management  effectiveness  in  software  projects  opens  up  many 
opportunities  for  improvement.  In  this  section,  we  introduce  four  approaches  for 
measuring  the  quality  of  software  project  management.  We  further  discuss  each  approach 
and  present  examples  of  the  existing  implementations.  The  significance  of  the  section  is 
guidance  for  the  development  of  project  management  effectiveness  metrics. 

B.  SUCCESS  PYRAMID 

Project  management  success  is  not  the  same  as  project  success  (Cooke -Davies, 
2002).  Even  though  most  practitioners  would  emphasize  that  software  project  success  is 
closely  related  to  project  management  quality  or  success,  there  is  no  established  empirical 
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evidence  for  such  a  relation  in  the  software  project  management  literature.  Related 
empirical  studies  in  the  software  engineering  field  or  even  in  the  project  management 
literature  are  quite  limited.  This  is  no  coincidence.  There  are  some  reasons  for  this. 

First,  even  though  there  are  many  studies  in  the  area  of  project  success  factors, 
there  is  no  established  criteria  for  project  success.  Pinto  and  Slevin  state  that  words  like 
success  and  failure  are  in  the  eyes  of  the  beholder.  They  also  emphasize  the  risk  of 
mislabeling  projects  as  success  instead  of  failure  or  vice  versa  without  a  well-established 
set  of  project  success  criteria  (Pinto  and  Slevin,  1998,  p.  379).  For  example,  Proccacino 
(2005)  investigated  how  various  practitioners  view  project  success.  His  study  adds  and 
introduces  another  view  to  existing  project  success  criteria.  White  (2006)  criticizes  the 
lack  of  suitable  measures  of  successful  projects.  Simply,  we  still  don’t  have  a  universally- 
accepted  definition  for  project  success.  How  then  can  we  relate  project  success  to  project 
management  success  when  there  is  no  clear  definition  for  project  success? 

Second,  there  is  no  theory  for  project  management  that  has  found  recognition 
(Smyth  and  Morris,  2007;  Pollack,  2007).  In  2006,  Turner  (2006,  pp.  1-3),  editor  of  the 
International  Journal  of  Project  Management,  wrote  a  series  of  editorials.  In  these 
editorials,  he  states  that  project  management  has  still  not  been  accepted  as  an  academic 
discipline.  He  concludes  that  one  of  the  reasons  for  this  is  the  lack  of  a  theory  for  project 
management.  In  that  and  following  editorials,  he  provides  a  normative  theory  of  project 
management  (Turner,  2006,  pp.  1-3,  93-95,  187-189).  In  2007,  Sauer  and  Reich  wrote  a 
response.  While  they  promote  the  idea  of  having  a  nonnative  theory  for  project 
management,  they  expressed  the  need  for  a  theory  that  helps  us  to  understand  the 
conditions,  constraints,  and  drivers  leading  to  functional  and  dysfunctional  behaviors 
(Sauer  and  Reich,  2007).  Therefore,  we  can  influence  such  behavior  to  reach  intended 
results.  While  theories  shape  a  discipline,  they  also  guide  researchers  to  investigate  the 
phenomenon.  As  a  result,  our  ability  to  develop  quality  criteria  for  project  management  is 
limited. 

Finally,  the  fields  of  software  engineering  and  project  management  are  quite 

young  when  compared  to  other  fields.  Research  works  related  to  foundations  of 

disciplines  take  time  to  build  up.  Reliable  empirical  studies  require  the  existence  of  a 
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certain  amount  of  fundamental  research.  Therefore,  our  ability  to  conduct  empirical 
research  in  the  field  of  software  project  management  is  limited. 

Defining  project  success  is  not  an  easy  task.  It  is  multifaceted  and  difficult  to 
measure  (Griffin  and  Page,  1996).  The  three  conventional  project  success  criteria  are 
time,  cost,  and  performance.  Pinto  and  Rouhiainen  (1998)  state  that  these  criteria  don’t 
work  in  the  modern  business  world.  The  tremendous  competition  in  this  modem  business 
world  requires  a  customer-oriented  focus.  Therefore,  customer  satisfaction  is  another  key 
criterion.  Glass  (1999)  points  out  the  need  for  a  new  theory  of  project  success.  Different 
stakeholders  may  have  different  concerns.  This  is  inevitable.  One  of  the  key  challenges  of 
any  project  management  is  to  align  the  goals  and  addressing  the  concerns  of  the 
stakeholders.  Linberg  (1999)  showed  that  the  definition  of  success  for  software 
practitioners  is  quite  different  from  the  conventional  criteria.  Software  practitioners  may 
classify  a  project  as  a  success  even  though  it  is  late  or  even  over  budget.  They  are  more 
concerned  with  the  quality  and  functionality  of  the  product.  In  addition,  they  may  even 
view  a  cancelled  project  as  a  success  due  to  the  lessons  learned  and  the  challenge  in  the 
project.  Agarwal  and  Rathod  (2006)  investigated  the  notion  of  software  project  success 
for  different  stakeholders.  They  examined  project  success  in  the  views  of 
programmers/developers,  project  managers,  and  customer  account  managers.  Procaccino 
(2002,  et  ah,  2005)  developed  a  quantitative  model  for  early  assessment  of  software 
development  success  in  the  practitioner’s  perspective.  Cooke-Davies  (2004a,  2004d) 
examines  the  issue  with  a  broader  view.  His  view  clarifies  some  challenged  research 
areas  beautifully.  He  provides  a  definition  of  success  at  different  levels.  His  questions  for 
each  level  help  us  to  focus  on  the  big  picture.  According  to  Cooke-Davies,  there  are  three 
levels  of  success: 

•  Level  1:  Project  Management  Success:  Was  the  project  done  right? 

•  Level  2:  Project  Success:  Was  the  right  project  done? 

•  Level  3:  Consistent  Project  Success:  Were  the  right  projects  done  right, 
time  after  time? 

These  levels  are  shown  in  a  pyramid  in  Figure  6.  The  figure  implicitly  implies 
that  the  success  of  each  level  depends  on  the  success  of  the  previous  level.  Even  though 
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this  is  the  fact  in  most  cases,  it  does  not  apply  in  all  cases.  The  figure  has  the  merit  of 
providing  an  overall  view  of  what  success  means  at  each  level.  It  is  possible  to  achieve  a 
successful  project  even  when  the  management  fails  or  vice  versa  (Munns  and  Bjeirmi, 
1996).  For  example,  even  though  the  management  has  done  a  good  job  in  completing  the 
project  within  budget,  on  time  and  with  the  expected  quality,  the  product  may  never  find 
its  share  in  a  competitive  market.  Then  the  fault  lies  on  the  executive  management  (or 
project  sponsor)  with  the  decision  to  undertake  such  a  project  delivering  a  product  that 
cannot  find  its  place  in  a  competitive  market.  In  that  case,  the  assumption  is  that  the 
project  management  team  is  handed  the  project  proposal  and  they  are  to  deliver  a  project. 


Project  \ 

Management 
Success 

/ Was  the  project  done 
right? 

Project  Success 

Was  the  right  project  done? 

Consistent  Project  Success 

Were  the  right  projects  done  right,  time  after  time  ?x 


Figure  6.  Success  Pyramid 

Munns  and  Bjenni  (1996)  provide  a  good  discussion  regarding  the  role  of  project 
management  in  achieving  project  success.  They  discuss  that  project  management  success 
suggests  a  shorter  term  while  project  success  has  a  longer  term.  This  is  consistent  with 
Cookie-Davies’s  view  of  success  at  different  levels.  As  a  result,  the  developed  framework 
for  success  at  different  levels  is  presented  in  Figure  7. 
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Figure  7.  The  Scope  of  Success  at  Different  Levels 


C.  DISCUSSION  OF  APPROACHES 

To  our  knowledge,  this  study  is  the  first  attempt  to  provide  a  framework  for 
measuring  the  effectiveness  of  software  project  management.  Related  measurement 
studies  in  the  project  management  literature  are  almost  non-existent.  The  management 
literature  focuses  on  organizational  effectiveness  that  is  remotely  related  to  project 
management  effectiveness. 
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We  have  identified  four  different  approaches  that  can  be  used  in  the  development 
of  methods  to  measure  the  effectiveness  of  software  project  management.  Figure  8  shows 
these  four  approaches  and  corresponding  metric  types.  Each  of  these  approaches  is 
discussed  in  the  following  sections. 


1.  Subjective  Evaluation 

In  this  approach,  the  project  participant’s  perception  is  used  in  the  evaluation  of 
the  project  management.  This  participant  may  be  the  project  manager,  the  technical 
manager,  or  the  developers.  Since  it  is  based  on  the  perception  of  the  participant,  this  is  a 
subjective  evaluation.  In  this  approach,  the  project  participant  is  simply  asked  to 
categorize  the  project  as  a  success/failure  or  rate  the  project  based  on  a  scale.  This 
approach  is  the  simplest  one  and  used  in  some  studies.  For  example,  Osmundson  et  al. 
(2003)  requested  the  project  managers  and  project  developers  rate  the  project’s  success 
based  on  a  scale  from  0  to  10  in  their  study.  In  another  study,  Verner  and  Evanco  (2005) 
investigated  the  project  management  practices  leading  to  success  in  in-house  software 
development.  They  analyzed  forty-two  successful  and  unsuccessful  projects  based  on  the 
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senior  software  practitioners’  categorization  of  their  projects.  In  his  doctoral  dissertation, 
Procaccino  (2002)  used  the  same  approach  and  his  study  is  based  on  the  view  of  software 
practitioners.  Gemuenden  and  Lechler  (1997)  conducted  an  empirical  analysis  on  448 
projects  to  detennine  critical  success  factors.  Their  study  relies  on  the  participants’  view 
as  to  whether  the  project  is  a  success  or  not. 

It  is  important  to  point  out  that  even  though  such  an  approach  is  subjective,  it  is 
hard  to  disregard  the  validity  (to  some  extent)  of  the  project  participant’s  perception.  The 
practitioners  have  a  sense  of  what  the  best  practices  are  and  if  those  are  followed  or  not. 
However,  as  Pinto  and  Slevin  (1998,  p.  357)  pointed  out  that  there  is  a  significant  risk  of 
mislabeling  a  project  as  a  success  or  failure  without  a  well-established  set  of  success 
criteria.  This  risk  is  more  significant  when  the  study  compares  the  successful  and  failure 
projects  based  on  the  subjective  evaluation  approach.  Because  when  the  project  is  in  fact 
a  failure  and  the  participant  mislabels  it  as  a  success,  then  this  evaluation  skews  both 
results  such  as  boosting  the  success  rate  and  decreasing  the  failure  rate. 

Another  important  consideration  is  that  the  measures  resulting  from  this  approach 
do  not  provide  any  insight  on  how  to  improve  the  management  of  the  project.  Just 
labeling  a  software  project  as  a  success  or  a  failure  without  understanding  the  causes  of  it 
has  limited  use  for  practitioners  and  researchers. 

2.  Questionnaire-based  Measurement 

In  this  approach,  the  measurement  of  management  effectiveness  is  based  on  the 
evaluation  of  responses  to  a  questionnaire.  Questionnaire-based  evaluations  are  common 
in  management  and  organizational  sociology  study  areas  (for  example  (Brown,  2003; 
(Baugh  and  Roberts,  1994;  Paul  and  Anantharaman,  2003;  Kinlaw,  1998;  Muller,  2003)). 
This  is  because  abstract  concepts  such  as  teamwork,  organizational  commitment, 
communication,  leadership,  etc.  are  hard  to  quantitatively  analyze.  This  approach  has 
been  used  in  the  development  of  a  quality  management  metric  for  software  development 
(Osmundson  et  al.,  2003). 

In  the  study  by  Osmundson  et  al.  (2003),  a  questionnaire  was  developed  to 

investigate  which  best  management  practices  are  followed  to  what  extend  in  a  software 
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project.  Then,  based  on  the  responses  to  the  questions,  the  quality  of  the  project 
management  is  measured.  They  also  compared  the  resulting  metric  (QMM)  with  a  metric 
gathered  via  subjective  evaluation  discussed  in  the  previous  section.  The  questionnaire 
investigates  four  important  areas  of  software  project  management.  They  are  requirements 
management,  project  planning  and  estimation,  risk  management,  and  people  management 
(Machniak,  1999).  People  management  is  further  divided  into  four  areas:  human 
resources,  leadership,  communication,  and  technical  competency  of  the  program 
manager.  The  complete  questionnaire  instrument  included  457  questions.  The  QMM 
metric  is  based  on  a  scale  from  0  to  10,  with  0  being  the  lowest  quality  score,  and  10 
being  the  highest  quality  score.  The  importance  of  the  QMM  study  is  the  focus  on  the 
development  of  a  metric  for  the  quality  or  effectiveness  of  project  management  in 
software  projects. 

COCOMO  II  incorporates  a  process  maturity  factor  (PMAT)  as  a  scale  factor  to 
the  effort  estimate  (Boehm  et  ah,  2000).  It  is  important  to  note  that  scale  factors  affect  the 
effort  estimate  exponentially.  In  COCOMO  II,  this  PMAT  factor  is  determined  using  one 
of  two  methods  (Clark,  2000).  The  first  method  is  based  on  the  SW-CMM  rating  of  the 
organization  when  there  is  one.  The  second  method  is  used  when  the  organization  does 
not  have  a  SW-CMM  rating.  The  second  method  uses  another  rating,  Equivalent  Process 
Maturity  Level  (EPML),  which  is  based  on  the  percentage  of  compliance  for  each  key 
process  area  goal  in  SW-CMM  model.  This  compliance  is  (EPML  rating)  evaluated  via 
the  responses  to  a  questionnaire  derived  from  eighteen  key  process  areas. 

3.  Metrics-based  Measurement 

Another  approach  for  measuring  the  effectiveness  of  software  project 
management  is  via  the  use  of  other  software  metrics.  For  example,  metrics  such  as  the 
number  of  defects  over  time,  software  complexity,  requirements  stability,  staff  turnover 
rate,  etc.  can  be  used  as  inputs  for  a  metrics  model  for  a  software  project  management 
effectiveness  metric.  This  type  of  measurement  is  in  fact  an  indirect  measurement.  When 
complex  attributes  are  measured  in  terms  of  simpler  sub-attributes,  this  measurement  is 
indirect  (Fenton,  1997).  Many  effort  predictions  use  several  levels  of  indirect 
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measurement  (Fenton,  1997).  Erdogmus  (2007)  presents  a  cost-effectiveness  indicator  for 
software  development.  He  uses  base  measures  such  as  nominal  output,  production  effort, 
rework  effort,  issue  count,  and  staffing  profile  to  derive  a  breakeven  multiple  as  an 
indicator  aggregating  productivity,  quality,  and  staffing  needs.  This  is  a  good  example  for 
this  approach  in  a  different  context.  Wohlin  and  Maryhauser  (2001)  provide  a  detailed 
method  for  assessing  software  project  success  using  subjective  evaluation  factors. 

To  our  knowledge,  there  has  not  been  an  attempt  for  the  development  of  a  metric 
for  assessing  the  management  effectiveness  of  software  projects  using  this  approach. 
Therefore,  we  provide  a  metric  model  for  such  measurement  to  guide  future  research. 
The  model  is  shown  below: 

n 

SPMEM  =  Measurement  _  function^ ^  wlml ) . 

i=i 

In  the  model  above,  m  is  a  metric  that  has  been  found  to  relate  to  the  metric  for 
management  quality,  which  is  denoted  by  SPMEM .  There  can  be  n  number  of  metrics. 
There  may  also  be  only  one  metric  and  in  that  case  n  equals  1 .  Examples  of  such  metrics 
may  include  programmer  productivity,  defect  reduction  rate,  certain  earned  value  metrics 
(EVM),  etc.  w.  is  the  weight  associated  with  a  certain  metric,  m. .  Such  weights  may  be 

required  since  different  metrics  may  relate  to  the  resulting  management  quality  metric 
differently.  Then  these  metrics  are  combined  via  a  measurement  function  based  on  the 
hypothesized  metric  model. 

A  generic  metric  model  was  presented  above.  Development  of  a  management 
effectiveness  or  quality  metric  for  software  projects  using  this  approach  requires 
significant  research  based  on  empirical  studies. 

4.  Model-based  Measurement 

In  this  approach,  the  metrics  for  effectiveness  or  quality  of  management  are 
derived  from  models  of  management  of  software  projects.  Currently,  this  approach  is  also 
conceptual  and  there  are  no  examples  implemented.  There  has  not  been  any  attempt  to 
measure  the  management  effectiveness  of  software  projects  based  on  a  model  of  project 
management. 
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For  quite  some  time,  researchers  have  been  focused  on  developing  software 
development  life-cycle  methodologies.  There  are  many  examples  of  methodologies  such 
as  waterfall,  spiral,  win-win,  rapid  prototyping,  agile  development,  SCRUM,  etc.  There  is 
also  a  field  called  software  process  research  within  the  software  engineering  discipline. 
Software  process  research  started  back  in  the  1980s  through  a  series  of  workshops  and 
events.  Due  to  many  software  application  failures,  researchers  focused  on  improving  the 
software  process.  The  assumption  is  that  there  is  a  direct  correlation  between  the  quality 
of  the  software  process  and  the  quality  of  the  software  application  developed.  A  good 
example  in  the  software  process  research  is  the  development  of  the  CMM  series  models. 
An  area  of  software  process  research  is  software  process  modeling.  There  are  a  number 
of  Process  Modeling  Languages  (PMLs)  developed  (Fuggetta,  2000).  Some  examples  are 
Process  Interchange  Fonnat  (PIF)  (Grunninger  et  ah,  1996;  1998),  Process  Specification 
Language  (PSL)  (Schlenoff,  Knutilla  and  Ray,  1998),  Unified  Process  Model  (UPM) 
(Kruchten,  1999),  Core  Plan  Representation  (CPR)  (Pease,  1998),  Workflow 
Management  Coalition  Process  Definition  (WfMC)  (1998),  and  Architecture  of 
Integrated  Information  Systems  (ARIS)  (Scheer,  1999).  A  review  of  these  PMLs  can  be 
found  in  (Breton  and  Bezivin,  2000). 

In  June  2005,  Business  Process  Management  Initiative  (BPMI)  and  Object 
Management  Group  (OMG)  merged  their  activities  and  formed  the  Business  Modeling  & 
Integration  (BMI)  Domain  Task  Force  (DTF).  They  have  developed  various  standard 
proposals  for  different  views  of  process  management,  such  as  Business  Motivation  Model 
(BMM)  specification  (OMG,  2007a)  and  Business  Process  Definition  Metamodel 
(BPDM)  (OMG,  2007b).  Even  Gantt  Charts  and  PERT  (Program/Project  Evaluation  and 
Review  Technique)  and  CPM  (Critical  Path  Analysis)  charts  are  process  models,  and  the 
development  of  Gantt  Charts  dates  back  to  1910s.  However,  there  is  a  significant 
difference  between  the  PMLs  mentioned  above  and  the  process  models.  While  the 
process  models  (such  as  Gantt,  PERT  and  CPM)  got  wide-acceptance  in  industry,  as 
Fuggetta  (2000)  pointed  out  few  (if  any)  of  the  proposed  PMLs  and  related  Process- 
centered  Software  Engineering  Environments  (PSEE)  have  been  transferred  into 
industrial  practice.  Fuggetta  states  that  the  goal  should  be  to  ease  the  adoption  of  PMLs. 
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Most  of  the  PMLs  are  heavily  technical  and  formal.  The  wide  adoption  of  Gantt,  PERT 
and  CPM  charts  tell  us  what  the  practitioners  would  like  to  see  in  these  types  of  process 
modeling  languages:  it  is  simplicity.  Since  these  PMLs  could  not  find  their  share  in 
practicality,  we  do  not  have  actual  project  data  based  on  models  developed  with  these 
languages.  Viable  effectiveness  measurements  for  software  project  management  require 
actual  data  from  projects,  which  we  do  not  have.  Process  models  are  developed  for  one 
specific  purpose  and  they  only  focus  on  one  aspect  of  project  management.  For  example, 
PERT  charts  are  used  for  prediction  of  the  project  schedule.  However,  managing  software 
projects  has  many  aspects. 

As  a  result,  Pinto  stresses  the  importance  of  modeling  the  business,  technical, 
financial,  environmental,  and  other  dimensions  of  the  project  before  committing  any 
significant  sources  or  even  before  the  go-ahead  (Pinto  and  Slevin,  1998,  p.  11).  Jaafari 
(2004)  provides  a  simplified  highest-level  representation  of  a  project  model  and  lists  the 
ideal  requirements  for  a  project  model.  He  stresses  that  we  still  have  a  long  way  to  go  in 
realizing  such  sophisticated  modeling  systems.  We  have  developed  a  simple,  visual  and 
formal  modeling  language  called  PROMOL  for  modeling  project  management  (Demir 
and  Osmundson,  2008).  This  modeling  tool  achieves  most  of  the  ideal  requirements  listed 
by  Jaafari.  According  to  Demir  and  Osmundson  (2008),  as  hypothesized,  there  are  two 
core  concepts  in  the  heart  of  project  management:  activities  and  entities.  These  two 
concepts  can  be  used  in  modeling  project  management.  Then,  the  quality  or  effectiveness 
of  these  activities  and  entities  in  a  project  management  model  can  be  used  as  inputs  for  a 
metric  model  for  effectiveness  of  project  management.  As  a  result,  a  high-level  metric 
model  may  be  formulated  as  follows: 

m  n 

SPMEM  =  Measurement  _  function^ ^  qat  +  ^  qe s )  . 

i= 1  7=1 

In  the  metric  model  above,  qaj  is  the  quality  of  an  activity  and  qet  is  the  quality  of 

an  entity.  These  activities  and  entities  are  components  of  a  project  management  model. 
There  can  be  m  number  of  activities  and  n  number  of  entities  in  the  model  that  is  of 
interest  as  inputs  for  the  SPMEM  metric  model.  The  measurement  function  is  a  function 
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that  combines  the  quality  measures  of  activities  and  entities.  This  function  is  specific  to 
the  metric  model  and  it  is  defined  in  the  metric  model.  Different  metric  models  may 
require  quite  different  measurement  functions.  It  is  important  to  emphasize  that  there  can 
be  a  number  of  variations  of  this  high-level  model.  Examples  of  these  variations  may  be 
where  a  model  is  including  only  activities,  including  only  entities,  or  basing  the  metric 
model  to  a  specific  life-cycle  development  model  and  deriving  the  activities  and  entities 
from  this  life-cycle  development  model. 

The  success  of  the  model-based  measurement  will  be  highly  dependent  on  the 
representation  capability  of  the  project  management  model.  When  these  project 
management  models  are  far  from  satisfactory,  then  the  resulting  metric  will  likely  be 
unsatisfactory. 

D.  CONCLUSIONS 

According  to  Evans,  Abela  and  Beltz  (2002),  the  first  characteristic  of 
dysfunctional  software  projects  is  the  failure  to  apply  essential  project  management 
practices.  This  is  derived  from  841  risk  events  in  280  software  projects.  480  out  of  841 
risk  events  (57%)  in  software  projects  are  due  to  not  applying  essential  project 
management  practices.  Jones  (2004)  reports  that  an  analysis  of  250  software  projects 
between  1995  and  2004  reveals  six  major  areas  effective  in  successful  projects  and 
inadequate  in  failing  projects.  They  are  project  planning,  project  cost  estimating,  project 
measurements,  project  milestone  tracking,  project  change  management,  and  project 
quality  control.  All  of  these  areas  are  related  to  software  project  management.  These 
studies  clearly  show  the  importance  of  project  management  in  achieving  software  project 
success.  Therefore,  project  management  metrics  are  the  key  to  rationally  focus  and 
substantiate  the  management  improvement  efforts. 

It  is  important  to  note  the  recognized  work  by  Basili  and  Rombaugh  (1988)  on  the 
Goal/Question/Metric  (mostly  known  as  GQM)  approach  for  development  of  software 
metrics.  They  provide  an  overall  approach  on  how  to  develop  metrics.  First,  it  is  very 
important  to  define  the  goal  of  the  measurement  activity.  This  sets  up  the  context  for  the 
measurement.  Second,  we  have  to  find  the  right  questions  for  identifying  the  metrics  that 
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are  going  to  be  used  in  the  measurement  effort.  Third,  we  have  to  choose  or  develop  the 
right  metrics  for  achieving  the  goal.  The  GQM  approach  is  completely  applicable  to  all 
the  approaches  presented  here.  The  goal  referred  in  GQM  is  already  defined  via  the 
context,  and  that  is  measuring  the  project  management  effectiveness  of  a  software 
project.  The  presented  four  approaches  help  us  to  refine  and  ask  the  right  questions.  The 
examples  and  high-level  models  presented  in  the  previous  sections  guide  us  in  identifying 
and  combining  the  necessary  metrics. 

In  this  doctoral  research,  two  approaches  are  employed  and  the  results  from  these 
approaches  are  compared  with  each  other.  These  are  the  subjective  evaluation  and 
questionnaire  based  measurement  approach.  Both  of  these  approaches  have  examples  in 
the  literature.  Therefore,  their  applicability  has  found  recognition  by  other  researchers  as 
well.  This  is  the  main  reason  for  the  selection  of  these  approaches.  According  to  the 
literature  review  conducted  by  the  author,  metrics-based  and  model-based  measurement 
approaches  have  not  yet  been  implemented  explicitly  for  evaluating  software  project 
management  effectiveness.  Earned  value  management  (EVM)  metrics  are  used  to 
monitor  project  performance.  EVM  metrics  may  be  used  with  reservations  for  assessing 
project  management  effectiveness  purposes.  For  example,  a  low  cost  performance 
indicator  (CPI=Earned  Value/Actual  Cost)  may  indicate  problems  related  to  project 
management;  however,  it  may  also  indicate  a  problem  with  project  cost  and  effort 
estimations.  Therefore,  EVM  metrics  are  limited  in  practice  for  this  purpose. 
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IV.  INTRODUCTION  OF  A  THEORY  OF  PROJECT 

MANAGEMENT 


A.  INTRODUCTION 

Even  though  project  management  has  been  a  recognized  practice  and  discipline 
for  many  years,  it  still  lacks  an  explicit  theory  of  project  management  and  sufficient 
theoretical  foundation  (Shenhar,  1998;  Turner,  1999;  Koskela,  2002;  Engwall,  2003; 
Smyth  and  Morris,  2007;  Pollack,  2007;  Jugdew,  2008).  Turner  (2006,  pp.  1-3),  the  editor 
for  the  International  Journal  of  Project  Management,  wrote  three  editorial  articles  in 
2006.  In  his  first  editorial,  he  clearly  stated  that  “...  there  is  not  yet  a  theory  of  project 
management.”  Therefore,  in  these  editorials,  through  a  series  of  premises,  corollaries  and 
lemmas,  he  built  a  structured  theory  of  project  management.  During  the  process,  he 
identified  a  number  of  inherent  properties  of  project  management.  Turner’s  project 
management  theory  helps  us  to  outline  a  framework  for  project  management  discipline. 
The  theory  helps  us  to  derive  study  areas  for  the  discipline.  Turner’s  theory  identifies  the 
domain,  defines  the  key  elements  and  constructs,  and  explains  the  relations  among  such 
constructs.  Some  of  the  premises,  corollaries  and  lemmas  he  provides  are  as  follows 
(Turner,  2006,  pp.  1-3): 

•  Premise  1.  A  project  is  a  temporary  organization  to  which  resources  are 
assigned  to  do  work  to  bring  about  beneficial  change.  (The  resources  may 
be  human,  material,  or  financial). 

•  Corollary  1.  Project  Contract  Management  and  Procurement  Management 
are  inherent  properties  of  project  management. 

•  Corollary  2.  Infonnation  Management  is  an  inherent  property  of  project 
management. 

•  Lemma  1.  A  project  consumes  resources,  particularly  financial  resources. 

•  Lemma  2.  A  project  produces  an  output  or  deliverable,  a  new  facility  or 
asset. 
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•  Lemma  3.  The  reason  the  owner  buys  the  asset  is  to  achieve  a  beneficial 
outcome. 

For  further  discussion  of  the  theory,  refer  to  (Turner,  2006,  pp.  1-3,  93-95,  187-189). 

Sauer  and  Reich  (2007)  provided  a  response  to  Turner’s  theory  of  project 
management  in  with  a  guest  editorial.  While  they  are  confirming  the  necessity  of  having 
a  theory  of  project  management,  they  also  raise  the  question  of  “what  kind  of  theory  we 
need.”  They  explain  the  normative  nature  of  Turner’s  theory  and  its  necessity.  However, 
they  also  express  that  Turner’s  theory  focuses  on  “what  should  be.”  Therefore,  the  theory 
doesn’t  explain  the  deviations  from  the  nonn,  the  effects  of  the  deviations  and  how  to 
correct  them.  Sauer  and  Reich  emphasize  the  need  for  a  theory  helping  us  to  understand 
the  conditions  and  drivers  leading  to  either  functional,  dysfunctional  or  both  behaviors. 
Such  theories  can  help  us  to  define  root  causes  and  create  a  change  for  the  desired 
outcome.  A  positive  theory  in  nature  can  satisfy  such  need  and  complements  normative 
theories. 

Due  to  the  normative  nature  of  Turner’s  theory  of  project  management,  the  theory 
does  little  to  enable  formal  analysis  of  projects  and  project  managements.  It  lacks  the 
definitive  power  to  statically  and  dynamically  investigate  the  inner  workings  of  a  project. 
The  development  of  a  new  project  management  theory  aims  to  satisfy  such  a  need.  The 
benefits  of  this  new  theory  include: 

•  It  simplifies  the  project  management  complexities  using  basic  concepts. 

•  It  has  explanatory  power  of  any  type  of  project.  It  is  not  restricted  to  any 
specific  domain  or  type. 

•  While  reducing  the  complexities  to  basic  concepts,  it  helps  us  to  formally 
define  and  analyze  projects. 

•  It  is  extendable,  and  therefore  lays  a  foundation  for  other  theories  to  build 
upon. 

The  theory  also  guided  us  to  develop  a  project  management  modeling  language 
(Demir  and  Osmundson,  2008;  Demir  and  Erguner,  2008).  This  modeling  language  is 
called  PROMOL.  The  language  is  supported  by  a  graphical  representation  to  ease  the 
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understanding  and  the  use.  The  applicability  and  scalability  of  PROMOL  in  modeling 
project  management  for  software  projects  is  analyzed  in  (Demir  and  Erguner,  2008; 
Erguner,  2008).  While  being  extendable,  the  produced  models  can  aid  us  in  static  and 
dynamic  analysis  of  projects.  It  is  possible  to  conduct  behavior  analysis  and  investigate 
project  management  best  practices  within  projects.  The  modeling  tool  enables  us  to  create 
project  histories  and  databases  to  enable  further  research  on  project  management. 
Overall,  this  new  theory  allows  us  to  gain  insights  about  projects  and  help  the  body  of 
project  management  knowledge  expand. 

B.  BASICS  OF  THE  THEORY  OF  PROJECT  MANAGEMENT 

The  theory  is  that  a  project  is  the  result  of  a  project  management  function,  which 
is  limited  over  a  specific  time.  The  inputs  for  this  function  are  a  limited  number  of 
activities  and  entities  related  to  any  part  of  the  project.  An  activity  is  a  named  process, 
function,  or  task  that  occurs  over  limited  time.  An  entity  is  something  that  has  a  distinct, 
separate  existence,  though  it  doesn’t  need  to  be  a  material  existence. 

It  is  important  to  note  that  a  project  is  the  sum  output  of  all  deliverables  as  well  as 
the  by-products  that  are  not  delivered  to  the  customer.  Examples  of  such  by-products  are 
patterns,  architectures,  methods,  reusable  components,  etc.  Notice  that  a  project  is 
whatever  the  project  management  function  generates.  However,  the  project  may  not  be 
what  the  customer  fully  or  partially  wanted.  Nonetheless,  the  project  management 
function  outputs  a  project  or  pieces  of  the  project. 

The  project  management  function  is  uniquely  described  by  activities  and  entities. 
Then  the  function  combines  and  transfonns  them  into  the  project.  This  function  is 
different  for  every  project,  assuming  that  no  two  projects  are  the  same.  All  stakeholders 
influence  this  function  by  negotiating  for  activities  and  entiies.  Then  the  optimum  for 
project  success  is  achieved  when  the  negotiations  are  pareto-efficient  (Langford,  2009). 
Basically,  this  project  management  function  may  be  viewed  as  an  ontology  of  activities 
and  entities  when  combined  transfonned  into  the  project. 

In  this  theory,  project  management  is  viewed  as  a  function  and  the  formulation  of 

the  project  management  function  is  given  below: 
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P  =  PM(alQ,a2Q,a3Q,...,amQ,el,e2,e3,...,en) . 


In  the  equation  above,  P  denotes  the  project  and  PM  is  the  project  management 
function  that  outputs  the  project.  The  inputs  of  the  project  management  function  are 
activities  denoted  by  a(),  and  entities  represented  by  e. 


Project  Management 


Activities  Entities 


Figure  9.  Activities  and  Entities. 

Note  that  both  activities  and  entities  are  not  unlimited  but  limited.  Therefore, 
there  exists  a  way  to  formulate  the  project  management  function.  Also,  it  is  imperative  to 
emphasize  that  activities  and  entities  are  distinctly  identifiable. 

Two  important  concepts  lies  in  the  heart  of  the  theory  as  depicted  in  Figure  9: 
activities  and  entities.  Examples  of  activities  are  requirements  analysis,  testing, 
stakeholder  analysis,  prototyping,  staff  meetings,  code  reviews,  etc.  Examples  of  entities 
are  project  manager,  staff,  teamwork,  test  cases,  leadership,  requirements,  documentation, 
etc.  Using  these  two  important  concepts,  it  is  possible  to  define  and  explain  any  project 
with  a  management  view  emphasis. 

C.  DISCUSSION  OF  REPRESENTATIONAL  THEORY  OF 

MEASUREMENT  AND  ITS  APPLICABILITY 

1.  Introduction 

Sauer  and  Reich  (2007)  stated  that  having  a  positive  theory  would  help  us  to 
understand  project  aberrations  and  improve  in  getting  better  results  by  identifying  the 
root  causes.  Previously,  a  new  theory  of  project  management  was  introduced.  However, 
by  itself  this  theory  is  not  sufficient  to  understand  and  measure  certain  phenomenon 
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within  the  project.  Therefore,  we  need  an  applicable  theory  of  measurement  that 
complements  the  theory  of  project  management. 

There  are  various  definitions  of  measurement.  In  Stevens  (1973),  it  is  defined  as 
“the  matching  of  an  aspect  of  one  domain  to  an  aspect  of  another.”  In  Sydenham  (1982), 
Fenton,  1994,  and  Fenton,  1997,  it  is  defined  as  “measurement  is  the  process  by  which 
numbers  or  symbols  are  assigned  to  attributes  of  entities  in  the  real  world  in  such  a  way 
as  to  describe  them  according  to  clearly  defined  rules.” 

There  are  only  a  few  theories  of  measurement  introduced  in  the  literature: 
classical,  operational  and  representational  theory  of  measurement  (Sarle,  1997; 
Sydenham,  1982). 

The  classical  theory  of  measurement  assumes  that  there  are  only  quantitative 
attributes  or  qualities  that  can  be  measured,  and  the  classical  approach  only  deals  with 
discovering  such  measures  and  attributes.  In  addition,  the  classical  theory  of 
measurement  assumes  existence  of  a  reality  that  is  being  measured.  The  classical  theory 
of  measurement  found  wide  applicability  in  physics  and  related  areas.  However,  it  was 
not  able  to  recognize  measurement  studies  in  social  and  behavioral  sciences. 

The  operational  theory  of  measurement  deals  with  the  definition  and  specification 
of  precise  measurement  operations.  On  the  other  hand,  it  avoids  the  assumption  of  the 
existence  of  a  reality  that  is  being  measured.  Its  concern  is  limited  with  the  operational 
aspect  of  measurement. 

The  representational  theory  of  measurement  handles  the  limitations  posed  by  both 
the  classical  and  operational  theory  of  measurement.  In  this  theory,  there  exists  a  reality 
that  is  being  measured  and  this  reality  may  also  be  one  that  is  not  readily  quantitative. 

Representational  theory  of  measurement  (Pfanzagl,  1968;  Krantz  et  ah,  1968; 
Sydenham,  1982;  Fenton,  1994)  is  found  to  be  applicable.  A  brief  discussion  of 
representational  theory  is  provided  as  it  is  pertinent  to  the  study.  It  is  a  brief  presentation 
taken  from  Sydenham  (1982). 

A  representational  theory  of  measurement  requires  four  parts: 
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•  An  empirical  relational  system  corresponding  to  a  quality. 

•  A  relational  system  based  on  a  defined  symbolism,  generally  it  is 
numbers. 

•  A  representation  condition. 

•  A  uniqueness  condition. 

2.  Empirical  Relational  System 

Let  ql,q2,...,qi,...  represent  the  individual  manifestations  of  some  quality  and 
define  Q  as  the  set  of  all  manifestations: 

Q  =  {qi,q1,...,ql,...}  . 

Define  Q  as  the  set  of  all  objects  that  we  are  interested  in  measuring: 

Q  =  {wuw2,...,  wt ,...}. 

There  exists  a  set  of  R  empirical  relations  rx,r2,...,^,...,rn  on  the  defined  set  Q. 
Define  R  as: 

R  =  {rl,r2,...,ri,...,rn}  . 

Then,  the  empirical  relational  system  is  represented  as: 

L  =  (Q,R). 

3.  Numerical  Relational  System 

Define  N  as  a  class  of  numbers  and  P  as  a  set  of  relations  on  N: 

P  =  {pl,p2,...,pi,...,pn) . 

So,  a  numerical  relation  system  is  represented  as: 

S  =  (N,P )  . 
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4.  Representation  Condition 


The  representation  condition  requires  that  there  exists  a  correspondence  between 
the  set  of  quality  manifestations  and  the  set  of  numbers  in  such  a  way  that  the  relations 
defined  on  the  set  of  quality  manifestations  is  preserved  on  the  other  set. 


Formally,  measurement  M  is  defined  as  an  empirical  operation: 

M-.Q^N  , 

such  that  L  =  {Q,R)  is  mapped  homomorphically  (structure -preserving  mapping)  onto 
S  =  ( N,P )  by  M  and  F.  One-to-one  mapping  is  denoted  by  F  with  domain  R  and  range  P: 

F:R^P. 

Therefore,  P  is  denoted  as: 

Pi=F(ri);pisP;rieR, 

where  p  is  an  n-ary  relation  if  and  only  if  it  is  the  image  under  F  of  an  n-ary  relation.  A 
homomorphic  mapping  is  that  for  all  r  e  R  and  all  p  e  P  and  p,  =  F(r) , 

^•0,, pi{M{ql),...,M{qi),-,M{qn))  . 

Measurement  M  is  not  a  homomorphism  (Sydenham,  1982)  since,  unlike  F,  M  is 
not  a  one-to-one  mapping.  There  can  be  mappings  to  the  same  number  because  there  may 
be  multiple  but  separate  qualities  corresponding  to  the  same  number. 

As  a  result, 

Y  =  (L,  S,M,F)  , 

where  Y  constitutes  a  scale  for  ni  =M(qj).  The  image  of  qt  in  N  under  M  is  called  the 
measure  of  q,  on  scale  Y. 
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5. 


Uniqueness  Condition 


There  may  be  multiple  mappings  for  which  the  representation  condition  is  valid. 
It  is  possible  to  have  transfonnations  from  one  scale  to  another  as  long  as  the 
representation  condition  is  valid.  The  uniqueness  condition  defines  the  class  of  scale 
transfonnations  to  mappings  for  which  the  representation  condition  is  valid  (Sydenham, 
1982). 

D.  DIRECT  AND  INDIRECT  MEASUREMENT 

There  are  two  methods  of  measurement:  direct  and  indirect.  In  Sydenham  (1982), 
direct  measurement  is  defined  as  the  method  “by  which  the  value  of  a  quantity  to  be 
measured  is  obtained  directly,  without  the  necessity  for  supplementary  calculations  based 
upon  a  functional  relation  between  a  quantity  to  be  measured  and  other  quantities  actually 
measured.”  The  key  difference  between  the  direct  and  indirect  method  is  obtaining  the 
measurement  with  or  without  the  necessity  of  measuring  other  qualities.  In  Sydenham 
(1982),  the  indirect  method  is  defined  as  the  method  “in  which  the  parameter  sought  is 
gained  by  use  of  intennediate  stages  of  different  units  which  are  linked  in  some  positive 
manner.”  Examples  of  direct  measurement  in  software  engineering  are  source  lines  of 
code,  duration  of  an  activity  such  as  testing,  number  of  defects,  and  effort  in  number  of 
man-hours  or  man-months.  Examples  of  indirect  measurement  in  software  engineering 
are  productivity  defined  such  as  number  of  lines  of  code  over  effort  in  man-month, 
requirements  stability  defined  as  number  of  requirements  at  start  over  total  number  of 
requirement  at  the  time  of  measurement,  etc. 

Since  our  goal  is  to  measure  effectiveness  of  project  management  in  a  software 
project,  the  complexity  of  the  concept  required  an  indirect  approach.  The  theory  helps  us 
measure  certain  properties  of  activities  and  entities.  New  concepts  are  also  introduced. 

Activities  and  entities  as  defined  in  the  previous  theory  of  project  management 
have  distinctly  identifiable  and  quantifiable  properties.  There  exists  a  way  to  quantify  and 
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measure  these  properties.  There  also  exists  a  way  to  combine  these  measures  and 
represent  them  as  another  measure  of  the  same  or  different  property  provided  that  a 
measurement  function  is  defined. 

Using  the  previous  theory  of  project  management  and  assumptions  provided 
above,  it  is  possible  to  derive  various  project  management  metrics.  For  example,  every 
activity  may  have  a  property  called  duration.  Assuming  a  complete  sequential  model  of 
the  project,  which  fully  orders  the  activities,  simply  adding  the  duration  properties  of  the 
project  activities  will  yield  the  duration  of  the  project.  Notice  that  the  resulting  type  of  the 
property  is  the  same  for  this  example. 

Properties  of  activities  and  entities  are  denoted  with  a  name  followed  by  a  dot 
after  the  activity  or  the  entity.  A  quality  exists  for  a  property  that  relates  the  property  to 
the  quality.  For  an  activity  “a()”,  a  property  “pr”  and  a  quality  “q,”  the  property  of  the 
activity  corresponds  to  the  quality  as  follows:  “a().pr=q.”  For  an  entity  “e,”  a  property 
“pr”  and  a  quality  “q,”  the  property  of  the  entity  corresponds  to  quality  as  follows: 
“e.pr=q.” 

Following  the  previous  definitions,  we  can  define  a  measurement  function  “F” 
for  a  property  “pr”  of  an  activity  “a()  ”  as  follows: 

Q  =  K().PrM)-P}Va20-P}2^^ 

q  =  aQ.pr 

where  “q”  is  the  quality  that  corresponds  to  the  property  “pr”  of  the  activity  “a  ()”  and 
“in  ”  and  “n”  are  the  identifiers  for  various  activities  and  entities.  Note  the  similarity  of 
the  formulation  with  the  PM  function  formulated  in  the  theory  of  the  project 
management.  Remember  that  PM  is  a  specially  defined  type  of  activity  in  the  theory. 

We  can  similarly  define  a  measurement  function  “F”  for  a  property  “pr”  of  an 
entity  “e”  as  follows: 

q  -  e.pr 
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where  q  is  the  quality  that  corresponds  to  the  property  “pr”  of  the  entity  “e.  ” 

Note  that  these  formulations  represent  the  most  general  form  in  which  the 
resulting  quality  is  a  combined  measure  of  the  different  properties  provided  that  the 
measurement  function  “F  ”  is  defined.  The  same  formulation  may  also  be  applied  when 
the  properties  of  the  activities  and  entities  are  the  same: 

q  =  K0.Pr  ia\  0-pr,  a2  Q.pr  ,...,am  Q.pr,  e,  .pr,  e2  .pr,...,en  .pr} 

q  =  Fe.Pr  {«! 0 -Pr, a2().pr,..., am  C ).pr , ex  .pr, e2 .pr, ...,en . pr }  ? 

where  “in  ”  and  “n  ”  are  the  identifiers  for  various  activities  and  entities. 

E.  A  THEORY  OF  PROJECT  MANAGEMENT  EFFECTIVENESS 
MEASUREMENT 

The  theory  of  project  management  effectiveness  measurement  lays  the  foundation 
for  the  development  of  the  software  project  management  effectiveness  metric.  Simply, 
we  can  assume  that  the  effectiveness  of  software  project  management  is  the  result  of  a 
measurement  function  in  which  the  inputs  are  the  effectiveness  properties  of  activities 
and  entities  used  as  inputs  in  the  project  management  function.  In  other  words,  when  we 
measure  the  effectiveness  of  activities  and  entities  in  a  project,  we  can  also  come  up  with 
the  effectiveness  of  project  management  using  a  measurement  function. 

PM.eff  =  FPMeff  {a{  0  .eff,  a 2  am  ().#,  e,  .eff,  e,  e/;  .eff) 

where: 

PM  :  The  project  management  function  for  a  specific  project,  P. 

PM.eff  :  The  effectiveness  property  of  the  project  management  function, 

PMt  for  project  P. 

FPMeff  :  The  measurement  function  defined  for  PM.eff  when  specific 

activity  and  entity  inputs,  a, ().eff, a, Q.eff',..., am Q.eff, e, .eff, e2 .efj\...,en .eff ,  are 

used. 

{a,(),a2() ,...,am0) :  The  activities  related  to  the  project 

{e1,e2,...,ej  :  The  entities  related  to  the  project. 
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V.  DEVELOPMENT  OF  A  FRAMEWORK  FOR  SOFTWARE 

PROJECT  MANAGEMENT 

A.  INTRODUCTION  TO  3PR  FRAMEWORK 

In  order  to  guide  the  development  of  the  software  project  management  metric,  it  is 
essential  to  be  able  to  frame  the  core  areas  of  software  project  management.  Therefore,  a 
framework  for  software  project  management  is  developed.  The  framework  is  quite  simple 
and  intuitive.  It  is  also  modifiable  to  suit  the  need  to  focus  different  areas  for  different 
types  of  projects.  First,  a  brief  overview  of  different  approaches  to  frame  the  project  and 
software  project  managements  will  be  presented. 

Project  Management  Institute’s  (PMI,  2004)  “Project  Management  Body  of 
Knowledge  Third  Edition”  (PMBOK  Guide),  identifies  five  project  management 
processes  groups: 

1.  Initiating  Process  Group. 

2.  Planning  Process  Group. 

3.  Executing  Process  Group. 

4.  Monitoring  and  Controlling  Process  Group. 

5.  Closing  Process  Group. 

According  to  the  PMBOK,  these  are  not  phases  of  a  project  and  they  may  be 
repeated  for  each  phase  where  appropriate.  PMBOK  also  identifies  and  lists  nine  project 
management  knowledge  areas: 

1 .  Proj  ect  Integration  Management. 

2.  Project  Scope  Management. 

3.  Project  Time  Management. 

4.  Project  Cost  Management. 

5.  Project  Quality  Management. 

6.  Project  Human  Resource  Management. 

7.  Project  Communications  Management. 

8.  Project  Risk  Management. 

9.  Project  Procurement  Management. 
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PMBOK  identifies  forty-four  project  management  processes  used  to  achieve  a 
project.  Appendix  E  provides  the  mapping  of  the  project  management  processes  to  the 
project  management  process  groups  and  the  knowledge  areas.  Even  though  this  mapping 
may  constitute  a  framework,  it  is  arguably  complex. 

Capability  Maturity  Model  Integration  version  1.1  (CMMI)  identifies  the 
following  process  areas  related  to  project  management  (CMMI,  2002): 

•  Project  Planning. 

•  Project  Monitoring  and  Control. 

•  Supplier  Agreement  and  Management. 

•  Integrated  Project  Management  for  Integrated  Product  and  Process 
Development  (IPPD). 

•  Risk  Management. 

•  Integrated  Supplier  Management. 

•  Quantitative  Project  Management. 

CMMI  version  1.1  prefers  to  divide  the  process  areas  into  two  process  area 
groups:  the  basic  project  management  process  areas  and  the  advanced  project 
management  areas.  Project  planning,  project  monitoring  and  control,  supplier  agreement 
and  management  are  addressed  as  basic  project  management  process  areas.  Integrated 
Project  Management  for  IPPD,  risk  management,  integrated  supplier  management  and 
quantitative  project  management  process  areas  are  categorized  as  advanced  project 
management  areas.  Figures  10  and  1 1  provide  the  interactions  among  these  process  areas 
for  basic  and  advanced  project  management  process  areas  respectively. 

As  it  is  observed  in  the  Figures  10  and  11,  the  interactions  among  these  process 
areas  are  not  easily  comprehensible,  even  though  only  certain  important  interactions  are 
depicted  in  the  figures.  CMMI  version  1.1  lists  requirements  management  and 
requirements  development  under  the  title  of  engineering  process  area,  and  configuration 
management  is  listed  under  the  title  of  support  process  area. 
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Figure  10.  Basic  Project  Management  Process  Areas  (Taken  from  (CMMI,  2002)) 


Process  Performance 
objectives,  baselines,  models 


Risk  exposure  due  to 
unstable  processes 


Figure  1 1 .  Advanced  Project  Management  Process  Areas  (Taken  from  (CMMI,  2002)) 
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The  “Program  Manager’s  Guide  to  Software  Acquisition  Best  Practices  Version 
2.31”  prepared  for  the  Software  Program  Managers  Network  (SPMN)  identities  nine 
principal  best  practices  (SPMN,  1998): 

1 .  Formal  Risk  Management. 

2.  Agreement  over  Interfaces. 

3 .  Formal  Insp  ections . 

4.  Metrics-based  Scheduling  and  Management. 

5.  Binary  Quality  Gates  at  the  Inch-Pebble  Level. 

6.  Program-wide  Visibility  of  Progress  vs.  Plan. 

7.  Defect  Tracking  Against  Quality  Gates. 

8.  Configuration  Management. 

9.  People-Aware  Management  Accountability. 

Also,  the  guide  groups  the  best  practices  into  seven  proven  management  areas: 

1 .  Risk  Management. 

2.  Planning. 

3.  Program  Visibility. 

4.  Program  Control. 

5.  Engineering  Practices  and  Culture. 

6.  Process  Improvement 

7.  Solicitation  and  Contracting. 

Every  management  area  contains  many  best  practices.  For  example,  risk 
management  has  five  best  practices,  planning  has  four,  program  visibility  has  four,  and  so 
on. 

In  (PMI,  2004;  CMMI,  2002;  and  SPMN,  1998),  process  areas  or  best  practices 
are  categorized  extensively.  Developing  a  framework  out  of  them  is  not  an  easy  task. 

Forsberg,  Mooz  and  Cottennan  (2005),  developed  an  elegant  visual  model  for 
project  management.  The  model  is  called  “the  wheel  and  axle  model,”  depicted  in  Figure 
12.  It  accounts  for  many  important  areas  of  project  management.  The  model  is  based  on 
five  essentials  for  every  project  (Forsberg  et  ah,  2005): 
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•  Organizational  commitment. 

•  Communication. 

•  Teamwork. 

•  Project  Cycle. 

•  Management  Elements. 


Figure  12.  The  Wheel  and  Axle  Model  [From  (Forsberg  et  ah,  2005)] 


The  visual  model  has  sequential  and  situational  practices.  The  phases  of  the 
project  cycle  are  sequential  and  the  management  elements  are  situational.  The 
management  elements  are  applied  throughout  the  project  cycle.  They  are  homogeneous  in 
this  aspect.  The  project  cycle  portrayed  as  an  axle  is  shown  in  Figure  13.  The  ten 
management  elements  are  presented  as  a  wheel  in  Figure  14 
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Figure  3.2  The  project  cycle  portrayed  as  an  axle. 


Figure  13.  The  Project  Cycle  Portrayed  as  an  Axle  [From  (Forsberg  et  al.,  2005)] 


Figure  3.1  Management  elements. 

Figure  14.  Management  Elements  [From  (Forsberg  et  al.,  2005)] 

The  project  cycle  has  three  periods:  study,  implementation,  and  operations.  The 
project  also  has  business,  budget  and  technical  aspects  managed  throughout  the  cycle. 
The  management  elements  are  depicted  as  the  spokes  of  a  wheel  and  they  are: 
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•  Project  Requirements. 

•  Organizational  Options. 

•  Project  Team. 

•  Project  Planning. 

•  Opportunities  and  Risks. 

•  Project  Control. 

•  Project  Visibility. 

•  Project  Status. 

•  Corrective  action. 

The  tenth  management  element  is  project  leadership  and  it  is  depicted  as  the  rim 
that  holds  the  spokes,  which  are  the  previously  listed  nine  items.  The  model  helps  us  to 
understand  various  important  elements  and  aspects  of  project  management.  It  also  helps 
us  to  visualize  the  interactions  among  the  elements  to  a  certain  level.  However,  the  model 
also  indicates  that  interactions  among  elements  and  processes  can  easily  get  complex. 

Philips  (2000)  identifies  three  key  perspectives  for  software  project  management: 
people,  business,  and  process.  He  emphasizes  that  having  these  perspectives  won’t  make 
a  project  successful,  but  it  will  help  to  go  a  long  way  to  making  success  possible.  He 
promotes  four  basic  principles  that  need  to  be  applied  with  discipline  and  perseverance: 

1 .  Balance  people,  process  and  product. 

2.  Promote  visibility. 

3.  Organize  by  using  configuration  management  tools  properly. 

4.  Use  standards  judiciously. 

Philips  highlights  that  all  undertakings  include  the  3Ps:  people,  process  and 
product.  In  successful  undertakings,  these  3Ps  are  managed  in  harmony.  Figure  15 
provides  Philip’s  mindmap  for  software  project  management. 
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Figure  15.  A  Mindmap  for  Software  Project  Management  [From  (Philips,  2000)] 

The  Software  Quality  Institute’s  Body  of  Knowledge  for  Software  Project 
Management  (SQI  BOK)  lists  thirty-four  competencies.  This  list  of  essential 
competencies  is  employed  by  the  most  successful  software  project  managers.  These 
competencies  are  categorized  into  three  parts:  Product,  Project  and  People  (Futrell, 
Shafer  and  Safer,  2002). 

Product  Development  Techniques 

1 .  Assessing  Processes. 

2.  Awareness  of  process  standards. 

3.  Defining  the  product. 

4.  Evaluating  alternative  processes. 

5.  Managing  requirements. 

6.  Managing  subcontractors. 

7.  Performing  the  initial  assessment. 
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8. 

Selecting  methods  and  tools. 

9. 

Tailoring  processes. 

10. 

Tracking  product  quality. 

11. 

Understanding  development  activities. 

Project  Management  Skills 

12. 

Building  a  work  breakdown  structure. 

13. 

Documenting  plans. 

14. 

Estimating  cost. 

15. 

Estimating  effort. 

16. 

Managing  risks. 

17. 

Monitoring  development. 

18. 

Scheduling. 

19. 

Selecting  metrics. 

20. 

Selecting  project  management  tools. 

21. 

Tracking  processes. 

22. 

Tracking  project  progress. 

People  Management  Skills 

23. 

Appraising  performance. 

24. 

Handling  intellectual  property. 

25. 

Holding  effective  meetings. 

26. 

Interaction  and  communication. 

27. 

Leadership. 

28. 

Managing  change. 

29. 

Negotiating  successfully. 

30. 

Planning  careers. 

31. 

Presenting  effectively. 

32. 

Recruiting. 

33. 

Selecting  a  team. 

34. 

Teambuilding. 
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Wiegers  (1996)  identifies  five  dimensions  of  a  software  project: 

•  Staff. 

•  Features. 

•  Quality. 

•  Cost. 

•  Schedule. 

Throughout  a  software  project,  the  listed  five  dimensions  have  to  be  managed. 
Figure  16  shows  these  dimensions.  These  dimensions  are  somewhat  dependent  on  each 
other;  the  relations  among  them  are  nonlinear  and  complex  most  of  the  time.  The 
dimensions  may  be  assigned  roles  on  a  project:  a  driver,  a  constraint,  or  a  degree  of 
freedom  (Wiegers,  1996).  The  driver  of  a  project  is  the  key  objective.  There  may  also  be 
multiple  drivers.  However,  if  all  dimensions  are  assumed  to  be  drivers,  there  is  no  point 
in  having  different  roles.  A  constraint  is  the  limiting  factor  for  the  project.  The  constraint 
has  to  be  outside  of  the  project  manager’s  control.  For  example,  a  fixed  cost  price,  where 
negotiation  with  the  customer  is  not  an  option,  is  the  constraint.  When  the  team  size  is 
fixed  and  the  manager  is  not  allowed  to  hire  new  team  members  or  detach  team  members 
from  the  project  organization,  then  staff  is  the  constraint.  The  rest  of  the  dimensions  that 
are  not  drivers  or  constraints  become  the  degrees  of  freedom.  When  the  project  manager 
has  control  over  adding  or  not  including  features,  then  the  feature  dimension  is  a  degree 
of  freedom. 


Figure  16.  The  Five  Dimensions  of  a  Project  [From  (Wiegers,  1996)] 
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Figure  16  presents  the  dimensions  on  a  kiviat  diagram.  Kiviat  diagrams  are  useful 
when  multiple  item  evaluations  are  presented  on  a  single  diagram.  A  kiviat  diagram  is  a 
polygon,  which  has  the  same  number  of  sides  as  the  number  of  variables.  Each  axis 
represents  a  data  category  and  different  scales  and  data  types  can  be  used.  However,  in 
this  case,  the  same  scale  will  be  used.  The  dimensions  are  categorized  with  respect  to  the 
flexibility  the  project  manager  has  over  the  dimension.  The  flexibility  of  the  dimension  is 
plotted  on  an  axis  of  the  kiviat  diagram.  The  scale  on  a  dimension  goes  from  zero 
flexibility  to  highest  flexibility  (0  to  10).  The  closer  the  plot  is  to  the  center,  the  less 
flexibility  there  is  for  that  dimension.  So,  for  a  complete  constraint  such  as  having  a  fixed 
number  of  team  members,  the  plot  on  the  staff  axis  would  be  the  closest  to  the  center. 
Figure  17  shows  the  flexibility  diagram  of  a  quality-driven  application.  The  plot  on  the 
quality  axis  is  closest  to  the  center.  As  the  diagram  shows,  the  project  manager  has  some 
flexibility  over  features  and  cost  while  having  considerable  flexibility  over  staff  and 
schedule. 

Understanding  the  driver,  the  constraint  and  degrees  of  freedom  in  a  project  and 
plotting  them  on  a  kiviat  diagram  helps  us  in  critical  decision  making  as  well  as  with 
prioritization. 


Figure  17.  Flexibility  Diagram  of  a  Quality-Driven  Application  [From  (Wiegers,  1996)] 
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According  to  Bach  (1995),  all  managers  are  faced  with  the  3Ps  while  developing 
software.  These  3Ps  are  people,  problem  and  process.  He  questions  whether  the  3Ps 
should  be  given  equal  weight  and  whether  one  should  be  given  more  focus  than  others. 
Bach  emphasizes  that  the  people  aspect  of  software  development  should  be  given  more 
focus  than  it  is  currently  given.  He  criticized  CMM  for  focusing  too  much  on  process 
rather  than  people  at  the  time  (1994).  One  year  later  in  1995,  the  first  version  of  the 
People  Capability  Maturity  Model  (P-CMM)  (Curtis  et  ah,  1995)  was  released  based  on 
the  work  by  Humphrey  (1989).  Later  the  work  was  called  Personal  Software  Process 
(Humphrey,  1996;  1997). 

Kulpa  (2007)  reports  an  interesting  graphic  from  a  CMM  introduction  class.  The 
graphic  presents  the  foundations  for  an  organization  and  referenced  them  as  quality 
leverage  points.  The  graphic  consists  of  a  three-legged  stool  figure.  In  the  graphic,  the 
stool  represents  the  organization.  The  legs  of  the  stool  are  people,  process  and 
technology.  She  points  out  the  reasons  to  use  People-CMM  in  her  article  (Kulpa,  2007). 
Figure  18  presents  the  graphic  mentioned. 


Figure  18.  Quality  Leverage  Points  [From  (Kulpa,  2007)] 
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Some  of  the  frameworks,  models,  perspectives,  standards  and  guidelines  for 
project  management  are  sampled  above.  Most  of  these  are  complex  in  nature  and 
arguably  complete. 

In  this  study,  a  simple  project  management  framework  is  developed  in  order  to 
accommodate  the  core  areas  of  project  management.  The  goal  was  to  identify  a  boundary 
for  project  management  in  which  we  can  easily  categorize  measurement  areas  for  project 
management.  This  framework  is  easily  modifiable  with  the  addition  of  new  areas  or  with 
the  removal  of  outdated  areas.  It  is  also  modifiable  in  the  sense  that  it  allows  the  focus  to 
be  different  areas  for  different  project  domains  and  types. 

The  framework  consists  of  four  main  areas  of  project  management: 

•  People. 

•  Process. 

•  Product. 

•  Risk. 

The  first  letters  of  main  areas  are  combined  and  the  framework  is  named  as  3PR. 
These  core  areas  help  us  to  partition  the  important  areas  of  software  project  management. 

B.  MAIN  PROJECT  MANAGEMENT  AREAS 

1.  People 

The  importance  of  people  management  in  project  development  efforts  is  quite 
well  established  (Brooks,  1995;  Bach,  1994;  Bach,  1995;  Philips,  2000;  Curtis  et  al., 
1995;  Curtis  et  al.,  2001;  Humphrey,  1995;  Humphrey,  1997;  DeMarco  &  Lister,  1999). 
Software  projects  are  developed  for  people  by  people.  The  people  area  especially  gets 
more  focus  in  a  software  development  environment,  since  the  development  is 
considerablly  human-intensive  compared  to  other  industries.  Kerzner  (1992)  provided 
some  classification  and  different  characteristics  of  projects  as  presented  in  Table  1. 
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In-house 

R&D 

Small 

Construction 

Type  of  Project/Industry 
Large  Aerospace/ 

Construction  Defense 

MIS 

Engineering 

Need  for  Interpersonal  Skills 

Low 

Low 

High 

High 

High 

Low 

Importance  of  Organizational 
Structure 

Low 

Low 

Low 

Low 

High 

Low 

Time  Management  Difficulties 

Low 

Low 

High 

High 

High 

Low 

Number  of  Meetings 

Excessive 

Low 

Excessive 

Excessive 

High 

Medium 

Project  Manager's  Supervisor 

Middle 

Top 

Top 

Top 

Middle 

Middle 

Management 

Management 

Management 

Management 

Management 

Management 

Project  Sponsor  Present 

Yes 

No 

Yes 

Yes 

No 

No 

Conflict  Intensity 

Low 

Low 

High 

High 

High 

Low 

Cost  Control  Level 

Low 

Low 

High 

High 

Low 

Low 

Level  of  Planning/Scheduling 

Milestones 

Milestones 

Detailed 

Detailed 

Milestones 

Milestones 

only 

only 

plan 

plan 

only 

only 

Table  1.  Classification  of  Projects/Characteristics  [From  (Kerzner,  1992)] 

In  Table  1,  there  are  two  categories  of  interest  related  to  this  study.  The  first  one 
is  the  aerospace/defense  industry.  It  is  quite  fair  to  assume  that  most  aerospace/defense 
industry  projects  rely  heavily  on  software  today  (Spruill,  2002).  The  second  one  is 
management  information  systems  (MIS). 

Therefore,  they  are  examples  of  software  projects  as  defined  in  this  dissertation. 
The  need  for  interpersonal  skills,  number  of  meetings,  and  conflict  intensity  are 
obviously  related  to  the  people  aspect  of  software  development.  In  aerospace/defense 
projects,  the  need  for  interpersonal  skills  and  conflict  intensity  is  high  and  the  number  of 
meetings  held  is  numerous.  In  contrast,  in  small  construction  projects  the  need  for 
interpersonal  skills,  conflict  intensity,  and  number  of  meetings  are  low.  While  this  shows 
important  differences  in  projects  from  different  industries,  it  also  stresses  the  importance 
of  the  people  aspect  in  aerospace/defense  projects. 

James  Bach  (1995)  takes  an  arguably  radical  position  in  what  aspect  needs  more 
focus  in  software  development  projects.  He  strongly  points  out  that: 

At  conferences  and  in  journals,  the  extraordinary  attention  we  give  to 
software-development  processes  is  misplaced.  Far  too  much  is  written 
about  processes  and  methods  for  developing  software;  far  too  little  about 
care  and  feeding  of  the  minds  that  actually  write  the  software.  Process  is 
useful,  but  it  is  not  central  to  successful  software  projects.  The  central 
issue  is  the  human  processor  -  the  hero  who  steps  up  and  solves  the 
problems  that  lie  between  a  need  expressed  and  a  need  fulfilled. 
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He  also  emphasizes  that  “I  argue  that  the  only  basis  for  success  of  any  kind  is  the 
heroic  efforts  of  a  dedicated  team.”  Even  though  his  views  might  be  seen  as  radical,  this 
may  be  the  result  of  resentment  due  to  lack  of  research  and  emphasis  on  people  issues  in 
software  development  when  compared  to  research  on  processes.  Weinberg  (1994)  says, 
“the  three  causes  of  failure  are  people,  people,  and  people.”  Again,  Thomsett  (1995) 
points  out  that  “most  projects  fail  because  of  people  and  project  management  concerns 
rather  than  technical  issues.”  Kulpa  (2007)  states  that  the  one  area  that  is  unaddressed  by 
organizations  is  the  people. 

Philips  (2000)  takes  a  more  central  approach.  He  stresses  the  importance  of 
having  a  balance  between  people,  process  and  product.  He  argues  that  the  road  to  success 
passes  from  hannonizing  these  3Ps. 

Brooks  (1995)  pointed  out  the  variations  in  programmer  productivity  as  a 
problem.  He  references  studies  reporting  an  order  of  magnitude  variations  dated  back  to 
1968  (Sackman,  Erickson  and  Grant,  1968).  DeMarco  and  Lister  (1987)  reported 
significant  computer  programmer  productivity  variations  ranging  from  one  to  ten  fold. 
Weinberg  (1994)  reported  variations  in  programmer  productivity  and  quality  from  twenty 
to  one.  Considerable  variations  exist  in  software  development  productivity.  Measuring 
programmer  productivity  is  not  trivial  (Spolsky,  2005).  It  is  very  hard  to  setup  an 
experiment  in  which  it  is  possible  to  control  every  factor  contributing  to  and  measuring 
the  productivity. 

In  one  of  the  most  widely-known  cost  estimation  technique,  COCOMO  II,  team 
cohesion  affects  the  effort  estimation  exponentially.  The  team  cohesion  scale  factor 
accounts  for  the  difficulties  in  synchronizing  and  managing  different  stakeholders 
including  users,  customers,  developers,  etc.  (CSE,  1999). 

Hughes  and  Cotterell  (2002)  point  out  that  people  with  practical  experience  in 
software  projects  will  clearly  state  an  important  aspect  of  software  project  management 
as  people. 

Project  Management  Institute’s  (PMI,  2004)  “Project  Management  Body  of 
Knowledge  Third  Edition”  (PMBOK  Guide)  lists  some  of  the  interpersonal  skills  needed 
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in  the  management  of  projects  in  the  areas  of  expertise  section.  The  list  includes  effective 
communication,  influencing  the  organization,  leadership,  motivation,  negotiation  and 
conflict  management,  and  problem  solving.  The  previous  PMBOK  edition  from  1996 
(PMI,  1996)  lists  all  of  the  above  except  motivation.  Even  though  PMBOK  recognizes 
the  importance  of  people  skills  in  the  management  of  projects,  it  doesn’t  go  into  detail  but 
instead  merely  lists  them. 

Given  the  many  evidences  of  the  importance  of  people  area  in  software 
development  projects,  inclusion  of  the  people  area  to  the  framework  is  essential.  The 
study  for  the  validation  of  the  framework  conducted  in  this  research  also  shows  that  the 
people  aspect  has  the  highest  importance  in  the  software  project  management  framework. 

2.  Process 

Without  a  defined  process,  gathering  a  bunch  of  practitioners  and  expecting  them 
to  work  in  harmony  for  a  common  goal  is  very  unlikely.  Two  things  may  happen:  either 
they  naturally  form  a  team  through  group  dynamics  and  even  setup  a  process  invisible  to 
the  outsider,  then  start  working  together  to  achieve  the  goal,  or  they  will  work  toward 
their  personal  ambitions.  In  other  cases,  where  there  is  a  defined  process,  practitioners  are 
assigned  to  or  voluntarily  fill  up  the  project  roles.  A  process  is  essential  to  the  project. 
Whether  the  process  is  effective  or  not,  or  the  process  is  well-defined  or  vaguely  exists, 
the  process  is  one  of  the  main  areas  of  project  management. 

IEEE’s  “Standard  Glossary  of  Software  Engineering  Terminology”  (IEEE,  1990) 
defines  the  process  as  “a  sequence  of  steps  performed  for  a  given  purpose;  for  example 
the  software  development  process.”  In  the  same  standard,  process  management  is  defined 
as  “the  direction,  control  and  coordination  or  work  performed  to  develop  a  product  or  a 
service.  Example  is  quality  assurance.” 

Within  the  framework,  the  main  area  of  the  process  encapsulates  the  focus  on  the 
various  key  processes  for  the  development  of  software  projects.  There  are  also  some 
other  key  processes  encapsulated  in  other  areas.  The  partitioning  is  based  on  whether  a 
process  intuitively  fits  the  main  area. 
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Two  of  the  most  widely  recognized  works  mainly  focus  on  processes.  CMMI  and 
earlier  various  CMMs  are  based  on  improving  the  maturity  of  organizations  by  improving 
their  processes  (CMMI,  2006).  CMMI  for  Development  versions  1.1  and  1.2  propose 
specific  and  generic  goals  for  each  identified  process  area.  As  previously  mentioned. 
Project  Management  Institute’s  (PMI,  2004)  “Project  Management  Body  of  Knowledge 
Third  Edition”  (PMBOK  Guide)  identifies  five  project  management  processes  groups: 
initiating,  planning,  executing,  monitoring  and  controlling,  and  closing.  Figures  19  and 
20  present  the  process  groups  and  their  interactions. 
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Figure  19.  High  Level  Summary  of  Process  Group’  Interactions  [From  (PMI,  2004)] 
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Figure  20.  Overview  of  Project  Management  Knowledge  Areas  and  Project  Management 

Processes  [From  (PMI,  2004)] 

Endres  and  Rombach  (2003)  present  Humphrey’s  law  as  “mature  processes  and 
personal  discipline  enhance  planning,  increase  productivity,  and  reduce  errors.”  As  a 
result,  inclusion  of  the  process  main  area  to  the  framework  is  essential. 

3.  Product 


According  to  2004  version  of  the  PMBOK  (PMI,  2004),  “a  project  is  a  temporary 
endeavor  undertaken  to  create  a  unique  product,  service,  or  result.”  In  the  framework,  the 
product  is  considered  as  the  outcome  of  the  project,  which  may  be  a  product,  service  or 
result.  This  view  is  also  shared  with  Philips’s  (2000)  definition  of  product:  “the  product  is 
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the  project’s  final  outcome.”  Products  include  software,  finnware,  documentation, 
reusable  artifacts,  training,  and  even  services  such  as  maintenance.  The  whole  purpose  of 
the  project  is  to  create  a  product  with  which  the  stakeholders  will  be  satisfied. 

The  most  important  characteristic  of  the  product  is  its  quality.  In  every  project, 
the  stakeholders  should  come  to  a  common  understanding  of  what  the  product’s  quality 
should  be.  The  earlier  this  common  understanding  is  reached  the  better  it  is.  According  to 
Blum  (1992),  there  are  two  views  of  quality:  internal  and  external.  While  internal  quality 
is  the  developer’s  view  of  the  software,  external  quality  is  the  stakeholders’  view  of  the 
software.  Internal  quality  includes,  but  is  not  limited  to,  efficiency,  testability, 
understandability,  and  modifiability.  External  quality  includes  usability,  correctness, 
reliability,  maintainability,  integrity,  etc.  It  is  preferable  to  make  these  quality  attributes 
as  measurable  as  possible;  however,  this  is  not  an  easy  task  in  every  project.  For  example, 
a  quality  attribute  such  as  usability  may  mean  different  things  for  the  developers  and  the 
users.  Thus,  it  is  essential  to  define  what  usable  means  as  early  as  possible  in  the  project 
development.  It  is  important  to  note  that  quality  is  not  a  feature  that  can  be  included  later 
in  the  product.  It  should  be  integral  to  the  whole  software  development  process. 

4.  Risk 

As  the  definition  of  the  project  stated  in  PMBOK,  a  project  is  undertaken  to  create 
a  unique  product,  service  or  result.  This  uniqueness  is  inherent  and  creates  a  certain 
amount  of  uncertainty  in  projects.  This  is  also  specifically  addressed  in  Turner’s  theory  of 
project  management  (Turner,  2006,  pp.  1-3,  93-95,  187-189).  In  this  theory,  lemma  4  and 
lemma  5  state  that  the  work  of  the  product  is  non-routine,  and  therefore  risky.  This  is  one 
of  the  inherent  aspects  of  projects.  Every  project  manager  or  project  management  team 
conducts  risk  management  activities  with  different  levels  of  rigor.  The  level  of  rigor 
varies  from  dedicated  formal  risk  management  procedures  to  ad  hoc  responses  to  risks. 

Risk  management  has  found  its  place  in  most  well-established  standards  and 
guidelines  such  as  PMBOK  (PMI,  2004),  CMMI  (2006),  program  manager’s  guide  to 
software  acquisition  best  practices  (SPMN,  1998),  Guide  to  the  Software  Engineering 
Body  of  Knowledge  (SWEBOK)  2004  Version  (IEEE,  2004),  INCOSE’s  (International 
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Council  on  Systems  Engineering)  Systems  Engineering  Handbook  version  3.1  (2003), 
NASA  Systems  Engineering  Handbook  (NASA,  2007),  and  the  Military  Standard  for 
Software  Development  and  Documentation  (DoD,  1995). 

Boehm  (1991)  points  out  that  in  most  software  project  disasters,  the  problems 
could  have  been  avoided  or  reduced  if  the  high-risk  elements  had  been  identified  and 
resolved  early  on  in  the  process.  Risk  management  practices  involve  two  primary  steps: 
risk  assessment  and  risk  control.  Risk  assessment  involves  risk  identification,  risk 
analysis,  and  risk  prioritization.  Risk  control  involves  risk  management  planning,  risk 
resolution,  and  risk  monitoring.  Capers  Jones  (1994)  identifies  an  alphabetic  listing  of 
sixty  risk  factors  in  his  novel  book.  This  book  is  a  good  source  of  information  for 
identification  and  resolution  of  risks  in  software  projects. 

Since  risk  management  is  an  inherent  aspect  of  projects,  software  project 
management  framework  includes  risk  as  a  main  area. 

C.  PEOPLE 

The  people  main  area  includes  seven  project  management  areas.  They  are 
communication,  teamwork,  leadership,  organizational  commitment,  project  manager, 
stakeholder  involvement,  staffing  and  hiring. 

1.  Communication 

Communication  can  be  generally  described  as  the  exchange  of  ideas,  opinions  and 
information  through  written  or  spoken  words,  symbols  or  actions  (Pearson  Education, 
2002).  A  successful  project  requires  constant  and  healthy  communication  between 
stakeholders.  The  importance  of  communication  in  project  development  is  well 
established  in  literature.  Among  all  of  the  project  management  areas  listed  in  PMBOK, 
communications  management  has  the  largest  impact  on  project  results  (Muller,  2003). 
Grinter  (1996)  expresses  that  good  communication  is  vital  to  establish  and  maintain 
control  over  the  software  development  process. 
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2. 


Teamwork 


Teamwork  may  be  defined  as  “the  concept  of  people  working  together  towards  a 
common  vision  or  a  goal  set  as  a  team.” 

3.  Leadership 

Leadership  may  be  defined  as  “the  ability  to  lead,  including  inspiring  others  in  a 
shared  vision.  Leaders  have  clear  visions  and  they  communicate  these  visions  to  their 
employees.  They  foster  an  enviromnent  within  their  companies  that  encourages  risk 
taking,  recognition  and  rewards,  and  empowerment  allowing  other  leaders  to  emerge” 
(Industry  Canada,  2008). 

4.  Organizational  Commitment 

Organizational  commitment  is  the  employee's  psychological  attachment  to  the 
organization  (Brown,  2003)  and  organizational  goals.  In  the  project  management  context 
and  in  this  framework  organizational  commitment  refers  to  the  commitment  to  project 
organization  and  project  goals.  There  is  an  important  difference  on  how  organizational 
commitment  is  viewed  in  this  framework  and  other  studies.  In  this  framework  especially, 
organizational  commitment  refers  to  commitment  from  all  stakeholders  including  project 
team  members.  In  most  other  studies,  organizational  commitment  is  viewed  from  the 
employee’s  view. 

5.  Project  Manager 

The  project  manager  position  is  a  key  role  in  project  organization.  The  project 
manager  is  mainly  responsible  for  planning,  directing,  controlling,  structuring, 
coordinating  and  motivating  in  the  project  organization.  In  this  study,  project  manager  is 
considered  as  a  role  and  authority  as  well  as  incorporating  the  personal  traits  within  the 
role.  The  role  includes  characteristics  of  both  a  good  manager  and  a  good  leader. 
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6. 


Stakeholder  Involvement 


Stakeholder  involvement  is  the  engagement  and  involvement  of  primary  and 
secondary  stakeholders  during  the  project  development  effort.  This  involvement  includes, 
but  is  not  limited  to,  planning,  decision-making,  development,  testing  and 
implementation  of  the  project.  For  a  successful  project  outcome,  stakeholder  involvement 
is  essential.  After  all,  the  project  is  undertaken  to  satisfy  the  needs  of  the  stakeholders. 

7.  Staffing  and  Hiring 

Staffing  may  be  defined  as  “the  practice  of  finding,  evaluating,  and  establishing  a 
working  relationship  with  future  colleagues  on  a  project  and  detaching  them  from  the 
project  organization  when  they  are  no  longer  needed.  Staffing  involves  finding  people, 
who  may  be  hired  or  already  working  for  the  company  (organization)  or  may  be  working 
for  competing  companies”  (Nation  Master,  2005).  Hiring  can  be  thought  to  be  within  the 
definition  of  staffing.  In  order  to  avoid  confusion  due  to  various  definitions  of  terms,  both 
terms  are  used  in  naming  the  area.  In  some  organizations,  hiring  means  employing 
project  team  from  outside  the  organization  and  staffing  means  employing  project  team 
members  within  the  organization’s  various  departments.  In  this  framework,  this  area  also 
includes  the  concept  of  placing  the  right  people  in  the  right  role. 

D.  PROCESS 

The  process  main  area  includes  four  project  management  areas.  They  are 
requirements  management,  project  monitoring  and  control,  project  planning  and 
estimation,  and  scope  management. 

1.  Requirements  Management 

“The  management  of  all  requirements  received  by  or  generated  by  the  project, 

including  both  technical  and  non-technical  requirements  as  well  as  those  requirements 

levied  on  the  project  by  the  organization”  (CMMI,  2006).  In  this  framework,  as  the 

definition  suggests,  requirements  management  is  the  management  of  requirements  and 

not  the  requirements  development  process.  This  is  an  important  distinction.  The 
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requirements  development  process  may  rely  on  a  specific  software  development  life 
cycle  model  such  as  waterfall,  spiral,  agile,  rapid  prototyping,  etc.  The  requirements 
management  process  itself  is  often  independent  of  the  life  cycle  development  model. 

2.  Project  Monitoring  and  Control 

Project  monitoring  and  control  are  actually  two  closely  related  project 
management  areas  combined  into  one  area.  Project  monitoring  is  the  process  of  keeping 
the  project,  project  related  factors,  and  project  metrics  under  continuous  observation. 
Project  control  is  the  process  of  ensuring  that  project  goes  according  to  what  is  planned  in 
the  project  plans  and  other  documentation.  In  addition,  the  project  control  process  ensures 
that  the  deviations  from  the  plan  are  kept  to  a  minimum  and  under  control. 

3.  Project  Planning  and  Estimation 

CMMI  1.2  defines  the  project  planning  as  follows: 

project  planning  includes  estimating  the  attributes  of  the  work  products 
and  tasks,  determining  the  resources  needed,  negotiating  commitments, 
producing  a  schedule,  and  identifying  and  analyzing  project  risks.  Iterating 
through  these  activities  may  be  necessary  to  establish  the  project  plan.  The 
purpose  of  project  planning  is  to  establish  and  maintain  plans  that  define 
project  activities  (CMMI,  2006). 

Even  though  estimation  is  included  in  the  previous  definition,  estimation  exists  in 
the  title  to  make  the  term  explicit  and  avoid  any  confusion.  Project  estimation  includes 
creating  and  establishing  estimates  of  project  cost,  schedule  and  necessary  resources 
using  various  methods,  techniques  and  tools. 

4.  Scope  Management 

In  simple  terms,  scope  management  is  the  process  of  defining  the  scope  of  the 
project  and  keeping  track  of  any  changes  of  the  scope.  It  also  includes  processes  to  limit 
the  changes  to  the  point  that  they  are  not  disruptive  to  the  success  of  the  project. 
According  to  the  project  management  challenges  survey  and  various  other  studies,  scope 
management  is  the  most  challenging  and  troublesome  area  in  projects. 
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E. 


PRODUCT 


The  product  main  area  includes  two  project  management  areas.  They  are 
configuration  management  and  quality  engineering. 

1.  Configuration  Management 

CMMI  1.2  defines  configuration  management  as: 

A  discipline  applying  technical  and  administrative  direction  and 
surveillance  to  (1)  identify  and  document  the  functional  and  physical 
characteristics  of  a  configuration  item,  (2)  control  changes  to  those 
characteristics,  (3)  record  and  report  change  processing  and 
implementation  status,  and  (4)  verify  compliance  with  specified 
requirements  (CMMI,  2006). 

Sometimes  the  meanings  of  configuration  management  and  scope  management  are  mixed 
among  software  practitioners.  However,  the  CMMI’s  definition  of  configuration 
management  clarifies  and  stresses  that  configuration  management  is  about  managing  the 
configuration  items.  These  configurations  items  include  intennediate  and  final  project 
artifacts  and  products.  Even  though  configuration  management  is  a  process  itself,  the 
focus  of  this  area  is  products.  Therefore,  configuration  management  is  placed  under  the 
product  main  area  to  avoid  confusion  due  to  definition  overload. 

2.  Quality  Engineering 

Quality  engineering  is  another  area  placed  under  the  product  main  area.  It  is 
important  to  note  that  the  term  quality  engineering  is  different  from  quality  assurance.  In 
many  organizations,  quality  assurance  is  used  to  refer  procedures  related  to  the  testing  of 
the  product.  In  others,  it  has  a  broader  meaning.  By  using  the  term  quality  engineering, 
the  framework  widens  the  area  and  includes  all  the  procedures  and  processes  conducted 
to  ensure  products  or  services  are  designed  and  produced  to  meet  or  exceed  customer 
requirements.  Quality  engineering  involves  all  activities  and  commitment  towards  the 
development  of  a  high  quality  product  to  meet  or  increase  the  stakeholders’  satisfaction. 
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F. 


RISK 


There  are  two  project  management  areas  listed  under  the  main  area  of  risk.  They 
are  risk  assessment  and  risk  control. 

1.  Risk  Assessment 

Risk  assessment  may  be  defined  as  “a  process  or  a  set  of  activities  that  involves 
identification,  analysis  and  prioritization  of  project  risks.”  In  some  projects,  risk 
assessment  is  conducted  with  quantitative  and  qualitative  fonnal  procedures  and 
techniques,  while  in  some  others  it  is  conducted  as  an  ad  hoc  process.  It  should  be  noted 
that  whether  it  is  formal  or  not,  the  quality  of  the  project  risk  assessment  also  depends  on 
the  skills  and  experiences  of  the  responsible  project  staff.  According  to  Boehm  (1991), 
risk  assessment  involves  risk  identification,  risk  analysis,  and  risk  prioritization. 

2.  Risk  Control 

Risk  control  may  be  defined  as: 

the  process  of  integrating  findings  from  the  risk  assessment  with  technical, 
financial,  policy,  and  non-technical  concerns  of  stakeholders,  to  develop 
and  select  suitable  risk  control  actions,  and  implementation  of  these 
actions.  Risk  control  actions  include  implementation  of  policies, 
standards,  procedures  and  physical  changes  (LesRisk,  2008). 

Risk  control  involves  risk  management  planning,  risk  resolution,  and  risk  monitoring 
(Boehm,  1991).  In  order  to  conduct  an  effective  risk  control,  an  effective  risk  assessment 
process  has  to  be  in  place. 

G.  CONCLUSION 

A  framework  for  software  project  management  titled  the  3PR  framework  was 
presented  in  this  chapter.  The  framework  consists  of  four  main  areas:  People,  Process, 
Product  and  Risk.  Fifteen  project  management  areas  were  identified  and  categorized 
under  these  main  areas.  First,  these  areas  were  identified  with  extensive  literature  search. 
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Then,  the  framework  was  validated  with  a  survey  study  with  the  participation  of  seventy- 
eight  software  practitioners  around  the  world. 

The  importance  of  the  framework  lies  in  its  simplicity.  It  establishes  the  main 
areas  in  software  project  management.  Every  activity  or  entity  that  is  related  to  project 
management  can  be  categorized  under  one  of  these  main  areas.  In  this  sense,  the 
framework  is  complete.  It  is  also  flexible  enough  to  represent  all  categories  and  types  of 
projects  with  different  focuses  on  different  main  areas.  The  software  project  management 
areas  categorized  under  the  main  areas  provide  guidance  for  project  managers  while 
allowing  them  to  focus  on  different  aspect  of  projects.  In  addition,  the  framework  guides 
researchers  in  developing  software  project  management  metrics. 
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VI.  VALIDATION  OF  THE  FRAMEWORK  FOR  SOFTWARE 

PROJECT  MANAGEMENT 

A.  INTRODUCTION 

A  survey  was  conducted  among  software  development  practitioners  in  order  to 
validate  the  framework  developed  earlier.  The  following  sections  provide  the  information 
regarding  the  survey  study,  methodology  and  results. 

B.  SURVEY  STUDY 

The  survey  study  is  an  important  part  of  the  research.  Many  approaches  were 
proposed  in  the  literature.  Some  of  them  are  listed  and  detailed  in  the  previous  sections. 
However,  only  few  of  them  are  widely  applied,  tested  and  empirically  supported;  the  rest 
of  them  are  based  on  the  views  and  experiences  of  various  research  and  practitioners. 
Therefore,  the  empirical  support  of  the  framework  is  an  important  contribution  of  this 
research. 

C.  PILOT  SURVEY  STUDY 

1.  Pilot  Study  Introduction 

Before  launching  the  full-scale  survey  study,  a  pilot  study  was  conducted.  Van 
Teijlingen,  Rennie,  Hundley  and  Graham  (2001)  stress  the  importance  of  pilot  studies. 
According  to  them,  the  tenn  pilot  study  refers  to  “mini  versions  of  a  full-scale  study  (also 
called  “feasibility  studies”),  as  well  as  the  specific  pre-testing  of  a  particular  research 
instrument  such  as  a  questionnaire  or  interview  schedule.”  Pilot  studies  are  an  important 
part  of  a  good  study  design.  Sometimes  pilot  studies  are  omitted  due  to  various  reasons. 
They  are  costly,  time-consuming,  and  they  consume  resources  otherwise  reserved  for  the 
full-scale  study.  However,  they  increase  the  likelihood  of  survey  study  success,  and  pilot 
studies  help  to  avoid  a  disaster  such  as  wasting  all  the  critical  resources  due  to  various 
design  errors. 
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The  reasons  for  completing  the  pilot  study  in  this  research  are: 

•  Guiding  the  development  of  the  research  design. 

•  Testing  the  research  design  and  the  instrument. 

•  Testing  of  the  surveying  technique  (whether  web-based  and  paper  based 
surveys  are  adequate,  or  if  structured  or  semi-structured  interviews  will  be 
needed). 

•  Understanding  and  forecasting  of  difficulties  for  the  full-scale  study. 

•  Testing  whether  the  population  sampling  method  is  viable. 

•  Testing  the  understandability  of  the  wording. 

•  Understanding  the  limitations  of  the  survey  study. 

•  Guiding  the  assessment  of  the  construct,  internal  and  external  validity. 

The  pilot  study  was  extremely  useful  in  this  case.  The  results  of  the  pilot  study  led 
to  modifications  and  enhancements  in  the  full-scale  study.  It  helped  uncover  some 
problems  regarding  the  surveying  protocol. 

Some  of  the  characteristics  of  the  pilot  study  for  this  research  are  listed  as 
follows: 

•  The  pilot  survey  instrument  and  research  design  followed  the  same 
principles  as  the  full-scale  study. 

•  The  participants  of  the  pilot  study  were  randomly  drawn  from  the  pool  of 
the  sampling  population  of  the  full-scale  study.  The  pilot  survey 
participants  were  not  used  again  in  the  study. 

•  The  data  collection  methodology  of  the  pilot  study  is  identical  with  the 
full-scale  study.  Both  the  pilot  study  and  the  study  used  a  self- 
administered  questionnaire.  The  questionnaires  had  two  versions.  One  is 
web-based  and  the  other  is  paper-based. 

2.  Pilot  Study  Instrument 

The  pilot  study  instrument  is  a  self-administered  questionnaire.  The  questionnaire 
consists  of  an  administrative  introductory  section  and  four  research  related  sections.  The 
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paper-based  version  of  the  survey  instrument  includes  six  questions.  A  copy  of  the  pilot 
survey  questionnaire  is  provided  in  Appendix  B.  The  web-based  survey  instrument  has 
eight  questions.  It  was  developed  using  a  commercial  surveying  tool  (SurveyMonkey, 
2007).  The  tool  utilizes  various  web  technologies  to  develop  quick  web  surveys.  Both 
versions  are  essentially  the  same.  The  only  difference  is  that  the  web-based  version  has 
the  administrative  sections  presented  as  questions.  The  first  two  questions  in  the  web- 
based  survey  instrument  are  used  for  the  administrative  section. 

In  the  pilot  study,  there  was  an  open-ended  question,  which  was  left  out  in  the 
full-scale  study.  The  goal  of  the  question  was  to  gather  the  participant’s  opinion  on  how 
to  improve  the  survey  instrument.  Valuable  insights  were  collected  from  the  feedback 
provided  via  this  question. 

3.  Pilot  Study  Results 

The  pilot  study  results  led  to  some  improvements  in  the  study.  The  results  and 
some  of  the  improvements  for  the  full-scale  study  are  listed  as  follows: 

•  Forty-four  survey  invitations  were  sent  out.  This  population  was  randomly 
selected  from  the  pool  of  the  total  sample  population.  There  were  twelve 
responses,  yielding  a  response  rate  of  27.7%.  This  rate  is  almost  the  same 
as  the  response  rate  in  the  full-scale  study.  The  responses  showed  that  the 
selected  population  is  the  right  population  for  the  study. 

•  One  of  the  feedbacks  indicated  the  necessity  of  a  glossary  section  for  the 
survey  to  eliminate  possible  misunderstandings.  Therefore,  a  glossary 
section  was  added  to  the  study. 

•  Two  of  the  participants  indicated  the  need  for  an  explicit  scale  for  the 
second  question  in  the  paper  version  of  the  survey.  Even  though  an 
explicit  scale  was  not  provided  for  this  question,  the  participants  were  able 
to  answer  the  question  without  difficulty.  A  scale  was  added  with  the 
question. 

•  Most  respondents  indicated  that  the  framework  proposed  was  sufficient. 
No  significant  improvement  was  suggested  for  the  framework. 
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•  Two  of  the  responses  specifically  indicated  that  all  areas  regarding  the 
software  project  management  were  covered  in  the  research. 

•  The  survey  length  was  found  to  be  reasonable. 

•  The  participants  found  the  questions  understandable. 

•  The  last  question  of  the  survey,  inquiring  about  possible  suggestions  to 
improve  the  survey,  was  deleted  in  the  full-scale  study,  since  this  question 
was  specifically  amended  for  the  pilot  study. 

•  The  analysis  of  the  responses  to  the  third  question  were  as  follows: 

o  People  =  39.16  % 

o  Product  =  18.33  % 

o  Process  =  25.00  % 

o  Risk  =17.50% 

o  The  same  ordering  with  similar  ratings  was  found  in  the 
full-scale  study. 

•  The  responses  to  the  second  question  of  the  pilot  study  were  analyzed  and 
the  ratings  were  ordered.  The  ordering  of  the  ratings  was  significantly 
similar  to  the  one  gathered  from  full-scale  study. 

•  Even  though  the  sample  size  was  quite  limited  for  the  pilot  study,  the 
analysis  of  the  responses  showed  that  the  responses  are  significantly  close 
to  the  responses  gathered  from  the  study.  This  may  be  the  result  of  a  good 
random  sampling  in  the  pilot  study. 

•  Asa  result,  the  survey  instrument  and  the  data  collection  procedures  were 
found  to  be  sufficient  with  the  necessity  of  a  few  modifications  and 
improvements. 

D.  SURVEY  INSTRUMENT 

A  copy  of  the  survey  instrument  is  provided  in  Appendix  C.  The  survey 
instrument  was  a  self-administered  questionnaire  and  contained  thirteen  questions.  The 
first  two  questions  were  needed  for  the  surveying  protocol.  In  the  third  and  fourth 
questions,  necessary  background  information  regarding  the  respondents  was  collected. 
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The  fifth,  eleventh  and  twelfth  questions  were  used  to  identify  the  importance  of  project 
management  areas  listed  previously.  The  sixth,  seventh,  eighth,  ninth,  and  tenth  questions 
were  used  to  identify  challenging  project  management  areas  in  software  projects.  In  the 
online  version  of  the  questionnaire,  the  order  of  the  choices  in  the  fifth  question  was 
randomized.  Such  randomization  eliminates  bias  due  to  ordering  of  the  choices. 

E.  TIME  FRAME  OF  THE  SURVEY 

The  timeframe  of  the  survey  study  is  the  first  quarter  of  2007.  The  survey  study 
took  around  four  months  in  total,  including  the  pilot  study. 

F.  POPULATION  OF  THE  SURVEY 

The  survey  invitation  was  distributed  to  over  four-hundred  software  development 
practitioners.  The  exact  number  of  invitations  that  reached  the  survey  sample  population 
is  not  known  because  a  portion  of  the  sample  population  is  from  Software  Development 
Forum  Software  Engineering  Management  Special  Interest  Group 
(SDFORUMSEMSIG).  An  invitation  was  posted  on  the  special  interest  group  web  page, 
where  the  number  of  members  was  increasing  every  day.  Therefore,  at  the  time  of  the 
start  of  survey  study,  it  is  assumed  that  the  posting  reached  around  170  members  of  the 
group  via  periodic  e-mail  messages.  Two-hundred  thirty-four  e-mail  invitations  were  sent 
to  software  development  practitioners.  This  sample  population  is  gathered  from  various 
sources  such  as  known  colleagues  and  references,  web  search,  and  authors  of  books  and 
articles  from  various  journals.  The  primary  qualification  criterion  was  having  software 
project  development  experience.  The  selection  of  the  sample  population  was  random. 

G.  ANALYSIS  OF  THE  RESPONSES  TO  THE  SURVEY 

•  There  were  104  responses  to  the  survey.  The  response  rate  is  around  26%. 

•  Two  of  the  104  indicated  that  they  don’t  want  to  be  included  in  the  study, 
so  their  responses  were  left  out. 

•  One  of  the  104  indicated  his  lack  of  experience  in  the  field,  and  therefore 
the  response  was  left  out. 
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•  Twenty-one  of  the  104  responses  were  incomplete,  therefore  unusable. 

•  There  were  around  eighty  valid  responses.  Only  a  few  of  them  were 
partially  usable  and  the  rest  were  completely  usable. 

Table  2  provides  the  number  of  responses  to  each  question. 


Question  # 

Total  Number  of  Responses 

Number  of  Valid  Responses 

1 

101 

101 

2 

104 

102 

3 

90 

78 

4 

92 

78 

5 

82 

78 

6 

81 

78 

7 

81 

78 

8 

81 

78 

9 

81 

78 

10 

81 

78 

11 

79 

75 

12 

72 

70 

13 

49 

47 

Table  2.  The  Number  of  Responses  to  Each  Question 


1.  Question  1 

This  question  was  used  to  record  the  identification  code  assigned  to  the 
prospective  survey  participant.  It  was  only  used  in  the  web-based  questionnaire  because 
the  commercial  tool  required  such  a  method.  For  the  paper-based  version,  it  was  already 
coded  in  the  survey  instrument  packet. 

Among  104  survey  respondents,  101  of  them  provided  the  identification  code  they 
were  sent.  These  codes  were  used  as  identifiers  and  to  keep  track  of  responses. 
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2. 


Question  2 


This  question  asked  for  consent  to  participate  in  the  study.  While  it  appeared  as  a 
question  in  the  web-based  survey  instrument,  there  was  a  separate  section  in  the  paper- 
based  version  in  which  the  section  wasn’t  assigned  a  question  number. 

Among  104  survey  respondents,  102  of  them  indicated  that  they  were  willing  to 
participate  in  the  study  by  responding  with  a  “Yes”  to  this  question.  Two  of  the  survey 
participants  indicated  their  unwillingness  for  participation  to  the  study. 

3.  Question  3 

Among  104  survey  respondents,  ninety  of  them  responded  to  this  question. 
However,  only  seventy-eight  of  the  participants  who  filled  out  this  question  participated 
in  the  rest  of  the  survey. 

This  question  inquired  about  past  work  experiences  of  the  survey  participants. 
The  respondents  provided  their  roles  in  software  projects  with  corresponding  experience 
in  years.  Figure  21  shows  these  responses. 
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Question  #3 


Roles  in  Software  Development 


Figure  2 1 .  Past  Roles  of  Survey  Participants 


4.  Question  4 

Question  4  simply  inquired  about  the  project  experience  of  the  survey 
participants.  The  goal  of  this  question  was  to  gather  background  data  on  respondents. 
Figures  22  and  23  show  a  graph  of  the  responses  by  response  count  and  percentage 
respectively. 
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Question  #4 


The  Number  of  Projects  Participated  In 


Figure  22.  Question  #4:  The  Number  of  Projects  Participated  In  -  The  Number  of  Responses 


Question  #4 


The  Number  of  Projects  Participated  In 


Figure  23.  Question  #4:  The  Number  of  Projects  Participated  In  -  Percentage 
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5.  Question  5 

This  question  was  one  of  the  key  questions  of  the  survey.  The  goal  of  this 
question  was  to  gather  the  opinions  of  software  development  practitioners  in  regards  to 
the  importance  of  certain  aspects  of  software  project  management.  The  respondents  were 
asked  to  rate  the  importance  of  a  particular  concept,  activity  or  role  within  software 
project  management.  The  rating  was  based  on  a  7-point  Likert  scale. 

The  response  count  for  this  question  was  78  out  of  104  respondents.  Two  more 
respondents  fdled  out  this  question;  however,  they  were  eliminated  due  to  lack  of 
experience  and  not  providing  adequate  background  infonnation.  For  each  item  in  the 
question,  the  mean  ratings  were  calculated.  Then  they  are  ordered  from  highest  to  lowest. 
Table  3  presents  the  ordering. 


Items  in  Question  #5 

Means  of  Ratings 

Communication 

5.69 

Teamwork 

5.41 

Leadership 

5.32 

Requirements  Management 

5.21 

Organizational  Commitment 

5.10 

Project  Manager 

5.09 

Stakeholder  Involvement 

5.05 

Project  Monitoring  and  Control 

5.01 

Project  Planning  and  Estimation 

4.99 

Scope  Management 

4.91 

Risk  Control 

4.86 

Staffing  and  Hiring 

4.82 

Configuration  Management 

4.81 

Risk  Assessment 

4.72 

Quality  Engineering 

4.64 

Support  Activities  (Training,  tools,  etc.) 

4.27 

Technical  Complexity 

4.17 

Table  3.  Software  Project  Management  Areas  and  Ordering  of  Ratings 
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One  of  the  choices  in  the  ratings  was  “No  Opinion.”  There  were  a  significantly 
low  number  of  “No  Opinion”  responses.  This  means  that  almost  none  of  the  respondents 
had  difficulty  in  associating  the  identified  areas  with  software  project  management.  This 
is  also  attributed  to  the  careful  design  of  the  question.  In  total,  there  were  eight  “No 
Opinion”  selections.  These  are  respectively  one  in  communication,  one  in  scope 
management,  one  in  staffing  and  hiring,  two  in  quality  engineering,  and  three  in  technical 
complexity. 

There  is  a  significant  finding  in  the  analysis  of  responses  to  these  questions.  The 
survey  participants  rated  six  of  the  software  project  management  areas  related  to  the 
people  dimension  among  the  top  seven  of  the  ratings.  This  is  also  a  confirmation  to  what 
will  be  found  later  in  question  11;  the  people  dimension  of  software  project  management 
rated  the  highest  among  other  dimensions.  Also,  process  dimension  related  areas,  which 
is  rated  the  second  highest  in  question  1 1 ,  are  found  to  be  among  the  second  highest 
ratings.  The  distinction  between  product  and  risk  related  areas  are  not  as  clear  as  people 
and  process  related  areas.  However,  it  is  also  important  to  note  that  the  ratings  are  very 
close  to  each  other  between  these  dimensions.  The  means  of  ratings  are  tabulated 
according  to  the  dimensions  and  presented  in  Table  4. 


People  Related  Areas 

Means  of  Ratings 

Communication 

5.69 

Teamwork 

5.41 

Leadership 

5.32 

Organizational  Commitment 

5.10 

Project  Manager 

5.09 

Stakeholder  Involvement 

5.05 

Staffing  and  Hiring 

4.82 

Process  Related  Areas 

Means  of  Ratings 

Requirements  Management 

5.21 

Project  Monitoring  and  Control 

5.01 
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Project  Planning  and  Estimation 

4.99 

Scope  Management 

4.91 

Support  Activities  (Training,  tools,  etc.) 

4.27 

Product  Related  Areas 

Means  of  Ratings 

Configuration  Management 

4.81 

Quality  Engineering 

4.64 

Technical  Complexity 

4.17 

Risk  Related  Areas 

Means  of  Ratings 

Risk  Control 

4.86 

Risk  Assessment 

4.72 

Table  4.  Means  of  Ratings 


6.  Question  6 


Question  6  was  the  first  question  of  the  third  part  of  the  survey.  This  and  the  next 
four  questions  constitute  the  entire  third  part  of  the  survey  study.  In  this  part,  the  focus  is 
the  respondents’  most  recent  project  experience.  The  question  simply  inquires  as  to  the 
size  of  the  project  in  terms  of  the  number  of  people  who  worked  on  the  project.  Figures 
24  and  25  show  a  bar  and  pie  chart  of  the  responses  respectively. 


Question  #6  -  Project  Size  in  Terms  of  Number  of 
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Figure  24.  Question  #6:  Bar  Chart  of  the  Responses 
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Question  #6-  Project  Size  in  Terms  of  Number  of 

Staff 

101 -or  more, 

6.4% 


Figure  25.  Question  #6:  Pie  Chart  of  the  Responses 


7.  Question  7 

The  goal  of  this  question  was  to  obtain  project  size  data  in  terms  of  source  lines  of 
code  (SLOC).  The  scale  is  divided  into  three  categories:  small  (less  than  20,000  SLOC), 
medium  (between  20,000  and  2  Millions  SLOC),  and  large  (more  than  2  Millions  SLOC). 
Even  though  this  categorization  makes  sense,  it  is  not  possible  to  draw  conclusions  about 
large  projects  due  to  the  limited  number  of  responses.  Figures  26  and  27  show  a  bar  and 
pie  chart  of  the  responses  respectively. 


Question  #7  -  Project  Size  in  Terms  of  SLOC 
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Figure  26.  Question  #7:  Bar  Chart  of  the  Responses 
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Question  #7  -  Project  Size  in  Terms  of  SLOC 

(large)  >2  Millions 


SLOC,  3.90% 


(small)  <20,000 
SLOC,  16.70% 
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□  (small)  <20,000  SLOC 


■  (middle)  20,000  SLOC  -  2 
Millions  SLOC 

□  (large)  >2  Millions  SLOC 


Figure  27.  Question  #7:  Pie  Chart  of  the  Responses 


8.  Question  8 

This  question  sought  to  identify  the  organization  type  in  which  the  project  was 
developed.  Figures  28  and  29  show  a  bar  and  pie  chart  of  the  responses  respectively. 
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Question  #8  -  Organization  Type 
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Figure  28.  Question  #8:  Bar  Chart  of  the  Responses 
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Figure  29.  Question  #8:  Pie  Chart  of  the  Responses 


9.  Question  9 

In  this  question,  the  application  type  developed  in  the  project  was  gathered.  There 
may  be  many  different  categorizations  of  software  applications.  However,  such  rigorous 
categorization  is  not  crucial  for  the  purposes  of  this  study.  Some  of  the  applications  carry 
characteristics  that  fit  more  than  one  type,  such  as  real-time  embedded  system,  or  web- 
based  database  application,  etc.  They  are  counted  in  both  categories  for  analysis 
purposes. 


Type  of  Application 

Response  Count 

Real-Time  Application 

30 

Web-Based  Application 

12 

Database  Application 

8 

Embedded  System 

7 

Various  Types  of  Management  Software 

5 

Distributed  System 

2 
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Various  Types  of  Applications  (such  as 

expert,  testing,  financial,  business  software 

etc.) 

9 

Various  Types  of  System  Applications  (such 

as  drivers,  IDE  extensions,  mainframe 

application,  integration  software  etc.) 

12 

Table  5.  Question  #9  :  Number  of  Responses  Categorized  Based  on  Type  of  Application 

Developed 


10.  Question  10 

In  this  question,  the  survey  participants  were  asked  about  the  management 
challenges  they  faced  in  their  last  project.  The  responses  gathered  via  this  question  are 
provided  in  Table  6.  The  goal  was  to  determine  if  there  is  a  change  in  the  trend  of 
challenges  faced  in  software  projects.  The  results  of  this  question  are  similar  to  previous 
studies.  The  conclusion  is  that  there  has  not  been  a  significant  change  in  the  trend  of 
challenges  faced  during  software  developments.  Therefore,  analysis,  findings,  and 
furthermore  the  assumptions  regarding  software  project  management  from  previously 
related  literature  is  still  applicable  for  this  research. 


Project  Management  Area 

Response 

Percentage 

Response 

Count 

Scope  management 

52.6  % 

41 

Requirements  management 

51.3% 

40 

Project  planning  &  estimation 

41.0% 

32 

Communication 

38.5% 

30 

Staffing  and  hiring 

33.3% 

26 

Project  monitoring  &  control 

28.2% 

22 

Risk  control 

26.9% 

21 

_ Technical  complexity _ 

_ 7.6.9% _ 

_ 2J _ 
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Stakeholder  involvement 

25.6% 

20 

Leadership 

25.6% 

20 

Configuration  management 

25.6% 

20 

Organizational  commitment 

24.4% 

19 

Quality  engineering 

23.1% 

18 

Teamwork 

21.8% 

17 

Risk  assessment 

19.2% 

15 

Project  manager 

14.1% 

11 

Other 

10.3% 

8 

Support  activities  (Training,  tools  etc.) 

9.0% 

7 

The  last  project  was  smooth  in  every. 

2.6% 

2 

Table  6.  Question  #10  :  Management  Challenges  in  Software  Projects 
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Question  #10 


Figure  30.  Question  #10:  Management  Challenges  in  Software  Projects 

11.  Question  11 

In  the  eleventh  question  of  the  survey  instrument,  participants  were  asked  to  rate 
four  main  project  management  areas: 

1.  People  (Project  Manager,  Staffing/FIiring,  Leadership,  Communication, 

Teamwork,  Stakeholder  Involvement,  Organizational  Commitment). 


108 


2.  Process  (Project  Planning/Estimation,  Scope  Management,  Project 
Monitoring  and  Control,  Support  Activities,  Requirements  Management). 

3.  Product  (Quality  Engineering,  Technical  Complexity,  and  Configuration 
Management). 

4.  Risk  (Risk  Assessment,  Risk  Control). 

The  total  rating  of  all  four  areas  should  add  up  to  100%.  There  were  seventy-five 
usable  responses  to  this  question.  The  mean  of  the  ratings  are  as  follows: 

People:  33.00% 

Process:  29.07% 

Product:  20.40% 

Risk:  17.53% 


Question  #1 1  -  Ratings  of  Main  Areas 


Risk,  17.53% 


Product, 

20.40% 


Figure  3 1 .  Mean  Ratings  of  Main  Project  Management  Areas 
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People 


Response 


Figure  32.  Distribution  of  Responses  in  People  Area 


Process 


Response 


Figure  33.  Distribution  of  Responses  in  Process  Area 
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Product 


Response 


Figure  34.  Distribution  of  Responses  in  Product  Area 


Figure  35.  Distribution  of  Responses  in  Risk  Area 
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12.  Question  12 


This  question  was  an  open-ended  question.  The  goal  of  this  question  was  to 
collect  the  participants’  view  on  the  most  important  aspects:  principles  or  practices. 

There  were  seventy  responses  out  of  104.  The  responses  are  categorized  using  a 
coding  method  referred  in  Seaman  (1999).  Tables  7  and  8  present  the  classification  of 
responses  to  software  project  management  (SPM)  areas  and  corresponding  frequencies. 
Table  9  provides  other  responses  left  out  in  the  categorization. 


Project  Management  Area 

Frequency 

Project  Planning  and  Estimation 

27 

Communication 

24 

Requirements  Management 

24 

Teamwork 

20 

Stakeholder  Involvement 

19 

Project  Monitoring  and  Control 

18 

Leadership 

16 

Other 

16 

Scope  Management 

14 

Organizational  Commitment 

13 

Staffing  and  Hiring 

10 

Project  Manager 

8 

Quality  Engineering 

7 

Risk  Control 

6 

Configuration  Management 

6 

Risk  Assessment 

6 

Support  Activities  (Training,  tools,  etc.) 

4 

Technical  Complexity 

0 

Table  7.  Classification  of  Responses  to  Question  #12  -  Sorted  from  Highest  Frequency 

to  Lowest 
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People  Related  Areas 

Frequency 

Communication 

24 

Teamwork 

20 

Leadership 

16 

Organizational  Commitment 

13 

Project  Manager 

8 

Stakeholder  Involvement 

19 

Staffing  and  Hiring 

10 

Process  Related  Areas 

Frequency 

Requirements  Management 

24 

Project  Monitoring  and  Control 

18 

Project  Planning  and  Estimation 

27 

Scope  Management 

14 

Support  Activities  (Training,  tools,  etc.) 

4 

Product  Related  Areas 

Frequency 

Configuration  Management 

6 

Quality  Engineering 

7 

Technical  Complexity 

0 

Risk  Related  Areas 

Frequency 

Risk  Control 

6 

Risk  Assessment 

6 

Table  8.  Classification  of  Responses  to  Question  #12  -  Rearranged  with  Respect  to  Main 

Project  Management  Areas 
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Other  Responses 

Frequency 

Need  balance  in  areas 

2 

Need  attention  to  design 

2 

Metrics  and  measurement  is  important. 

2 

Sponsorship  is  important. 

1 

Testing  is  important. 

1 

Being  open  to  various  technical  solutions 

1 

Technical  part  is  easy. 

1 

Follow  CMMI 

1 

Managing  heroes  at  work 

1 

Consideration  of  technical  aspects 

1 

Lessons  learned 

1 

Different  organizations  require  focus  on 

different  areas. 

1 

Consideration  of  systems  architecture  and 

systems  approach 

1 

Table  9.  Other  Responses  to  Question  #12 


A  quick  overview  of  the  responses  will  yield  that  quite  a  significant  portion  of  the 
responses  were  covered  with  software  project  management  areas  inquired  in  question  5. 
This  is  a  strong  indication  that  the  proposed  framework  is  valid  and  provides  good 
coverage. 

It  is  observed  that  there  are  some  minor  differences  with  the  ordering  of  ratings 
derived  from  responses  to  question  5  and  the  ordering  of  frequencies  derived  from  the 
categorization  of  responses  to  this  question.  There  may  be  a  few  reasons  for  these 
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differences.  First,  the  coding  technique,  reducing  similar  issues  to  one  category, 
inevitably  causes  some  data  loss.  Second,  the  factor  of  unresponsive  participants  may 
have  played  a  role  in  the  differences.  Not  all  the  participants  who  responded  to  question  5 
responded  to  this  question.  Also,  the  ratings  gathered  from  question  5  may  include 
insignificant  statistical  analysis  errors  that  lead  to  differences.  However,  the  obvious 
overlap  in  responses  to  both  questions  is  significant  for  this  research.  It  validates  the 
framework.  The  responses  to  both  questions  help  assess  the  internal  validity  of  the  survey 
study. 


13.  Question  13 

This  question  was  an  open-ended  question.  The  goal  of  this  question  was  to 
collect  the  survey  participants’  feedback  on  the  issues  that  are  mentioned  in  the  survey 
instrument.  Because  of  the  variance  in  the  responses,  a  coding  method  was  not 
successfully  applied. 

•  There  were  specific  responses  indicating  that  the  survey  instrument  has 
good  coverage. 

•  There  were  quite  a  number  of  responses  reemphasizing  some  of  the  areas 
already  mentioned  in  the  survey.  They  may  be  listed  as  politics,  teamwork, 
human  side  of  software  development,  importance  of  leadership, 
importance  of  risk  management,  and  project  championship  (such  concept 
is  implicitly  covered  in  the  area  of  stakeholder  management  such  as 
project  champion  is  a  stakeholder). 

•  There  were  some  responses  indicating  the  importance  of  measurement  and 
process  improvement  activities  in  software  projects. 

•  In  a  few  responses,  it  is  mentioned  that  it  is  difficult  to  separate  different 
aspects  of  project  management  listed  in  the  survey.  Most  areas  depend  on 
each  other. 

•  Two  of  the  respondents  indicated  the  importance  of  security 
considerations  during  project  development  such  as  software  assurance  and 
information  security. 
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•  In  a  few  responses,  the  importance  of  systems  thinking  is  emphasized.  The 
respondents  indicated  that  the  software  component  is  part  of  a  system  and 
eventually  the  software  development  effort  has  to  integrate  to  a  bigger 
system  development  effort. 

•  In  a  few  responses,  the  respondents  suggested  investigating  the  link 
between  different  software  life  cycle  development  approaches  and  the 
project  management  areas  covered  in  the  survey  study. 

•  Others  respondents  indicated  the  importance  of  requirements  activities, 
creating  and  visiting  lessons  learned  documents,  the  use  of  tools,  the 
negative  effects  of  task  switching  and  multitasking,  the  importance  of 
project  effort  estimation,  project  monitoring  and  control,  iterative 
development,  reuse,  significance  of  having  adequate  testing  facilities, 
project  monitoring,  the  importance  of  developer  feedback  in  project 
planning  efforts,  protecting  the  project  team  from  counterproductive 
external  interference,  and  system  safety  issues. 

H.  VALIDITY  OF  THE  SURVEY  STUDY 

The  validity  of  the  study  is  discussed  here.  This  study  was  conducted  as  a 
descriptive  study.  In  descriptive  studies,  the  researcher  merely  observes  the  events  and 
there  is  no  intervention.  Descriptive  studies  are  observational  in  nature,  and  hence  they 
are  also  called  observational  studies.  In  this  study,  we  asked  the  survey  respondents’  view 
on  identified  project  management  areas.  Project  management  challenges  they  have  faced 
in  their  last  projects  were  also  gathered.  We  did  not  intervene  in  their  projects  and 
therefore  affect  their  views.  In  most  research  experiments,  researchers  apply  a  controlled 
event,  method  or  procedure  to  understand  the  relations  between  dependent  and 
independent  variables.  Thus,  there  is  an  intervention  by  the  researcher.  This  intervention 
increases  the  complexity  of  the  study,  which  in  turn  raises  many  validity  concerns.  As  a 
result,  the  researchers  must  be  careful  in  these  experiments.  Most  validity  concerns  do 
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not  apply  to  descriptive  studies.  For  example,  internal  validity  is  only  relevant  when  the 
researchers  try  to  establish  casual  relations.  Therefore,  here  only  external  validity  related 
issues  are  briefly  addressed. 

External  validity  refers  to  the  validity  with  which  a  casual  relationship  can  be 
generalized  across  persons,  settings,  and  times  (Emory,  1980). 

The  survey  instrument  was  distributed  to  the  practitioners  from  different 
geographical  regions.  These  regions  include  North  America,  South  America,  Europe  and 
Asia.  There  were  no  responses  from  Australia  even  though  survey  invitations  were  sent  to 
practitioners  located  in  this  region.  It  was  observed  that  there  are  no  significant 
differences  among  practitioner  views  from  different  regions  of  the  world. 

The  survey  study  was  conducted  in  the  first  quarter  of  2007.  There  are  other 
survey  studies  reported  in  the  literature.  The  survey  results  are  similar  to  the  other  survey 
study  results  conducted  earlier. 

The  survey  participants  may  be  divided  into  two  categories  based  on  the  roles 
they  had  in  software  projects.  The  first  category  is  project  managers,  while  the  second 
category  is  developers.  The  responses  from  the  practitioners  in  these  two  categories  are 
similar,  especially  the  responses  to  question  1 1 .  Overall,  there  were  no  significant 
differences. 

The  sample  size  in  this  study  can  be  categorized  as  medium  compared  to  other 
survey  studies  conducted  on  the  topic.  Random  sampling  was  utilized.  When  the 
sampling  method  is  appropriate,  even  small  samples  will  provide  reliable  results.  The 
responses  were  continuously  monitored  during  the  study.  The  responses  to  the  question 
11  did  not  significantly  differ  when  the  sample  size  was  5,  10,  20,  50  and  78.  This  is 
attributed  to  the  quality  of  the  sampling. 

I.  CONCLUSION 

The  objectives  of  this  survey  study  were  to  identify  (i)  the  importance  of  various 
project  management  areas  and  (ii)  project  management  challenges  in  software  projects. 
For  the  purposes  of  this  research,  the  survey  study  reached  its  goals.  The  importance  of 
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project  management  areas  in  software  projects  has  successfully  been  identified.  This 
identification  led  to  the  conclusion  that  the  software  project  management  framework 
proposed  in  the  previous  section  is  valid.  The  results  of  this  survey  study  guided  the 
development  of  the  software  project  management  evaluation  instrument  and  evaluation 
model. 

The  survey  results  indicate  that  the  differences  in  the  importance  ratings  of  the 
main  areas  (people,  process,  product  and  risk)  are  distinct.  However,  that  is  not  the  case 
for  the  project  management  areas  listed  in  question  5.  The  survey  results  showed  that 
even  though  it  is  possible  to  rank  project  management  areas  based  on  their  importance, 
the  differences  between  the  ratings  of  project  management  areas  are  small. 
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VII.  SOFTWARE  PROJECT  MANAGEMENT  EVALUATION 
INSTRUMENT  (SPMEI)  AND  SOFTWARE  PROJECT 
MANAGEMENT  EVALUATION  MODEL  (SPMEM) 

A.  INTRODUCTION 

A  theory  of  project  management  presented  in  this  research  and  the  3PR 
framework  for  software  project  management  guided  the  development  of  the  instrument 
and  the  evaluation  model.  The  instrument  and  the  evaluation  model  design  was  a  major 
task  in  this  research.  While  half  of  the  research  effort  was  focused  on  building  the 
necessary  theoretical  foundation  for  this  research,  the  other  half  of  the  effort  was  focused 
on  the  development  of  the  instrument,  the  development  of  the  evaluation  model  and 
conducting  survey  studies  on  software  projects  to  investigate  the  use  and  applicability  of 
the  metric.  It  took  more  than  fifteen  months  to  develop  the  SPMEI  and  the  evaluation 
model.  The  main  goal  of  SPMEI  is  to  gather  data  on  what  happened  during  the  project 
development.  The  instrument  is  responded  to  as  such. 

B.  SOFTWARE  PROJECT  MANAGEMENT  EVALUATION  INSTRUMENT 
(SPMEI) 

1.  Basic  Characteristics  of  the  Instrument 

Basic  characteristics  of  software  project  management  evaluation  instrument 


(SPMEI)  are  provided  below  in  Table  10. 


Name  of  the  Instrument 

Software  project  management  evaluation  instrument 

Acronym 

SPMEI  (The  first  letters  of  the  words  in  the  name) 

Main  Use  of  Instrument 

To  get  data  on  what  happened  during  the  project 

development 

Type  of  Instrument 

Self-administered  Questionnaire 

Who  may  use  it? 

Executive  managers  overseeing  projects 

Project  managers 
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Project  technical  managers 

Process  improvement  or  metrics 

experts/ engineers 

Project  team  leaders 

Project  team  members  who  has  extensive 

knowledge  in  all  aspects  of  the  project 

Applicability 

Software  development  projects 

Software-intensive  development  projects 

Applicable  to  any  project  organization  size 

Applicable  with  any  software  development 

life-cycle  model 

Applicable  to  project  phases  after  some 

requirements  development  activities  are 

conducted 

Scope 

Project  start  to  project  delivery  (Project  start  is  the 

time  when  the  business  decision  is  made) 

Number  of  Sections 

15 

Number  of  Questions 

330-335 

Type  of  Questions 

Multiple  choice 

Statements  with  a  psychometric  scale  (5- 

point  Likert  item  based  on  agreement  to  a 

statement) 

All  questions  are  closed  form 

Time  to  complete 

2-3  hours 

Table  10. 

Basic  Characteristics  of  SPMEI 
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SPMEI  is  designed  as  a  self-administered  questionnaire  consisting  of  fifteen 
sections.  Each  section  corresponds  to  a  project  management  area  in  the  3PR  framework 
with  the  same  name.  While  collecting  data  for  survey  studies  in  this  research,  another 
section  was  included  to  collect  basic  data  about  the  projects  such  as  the  cost  of  the 
project,  the  number  of  people  involved  in  the  project,  the  length  of  the  project,  etc.  In 
Appendix  F,  a  copy  of  the  instrument  is  provided.  Each  question  in  SPMEI  inquires  about 
the  effectiveness  of  an  activity  or  an  entity  related  to  project  management. 

SPMEI  includes  330  to  335  questions.  Depending  on  the  characteristics  of  the 
project,  the  participant  responds  to  the  appropriate  questions.  Table  11  presents  the 
number  of  questions  in  each  section  of  SPMEI.  Table  12  provides  the  number  of 
questions  in  SPMEI  categorized  by  the  corresponding  main  area. 


Project  Management  Area 

Number  of 

Communication 

23 

Teamwork 

30 

Leadership 

17 

Organizational  Commitment 

26 

Project  Manager 

27 

Stakeholder  Involvement  (Market  or  Contract) 

12  or  16 

Staffing  and  Hiring 

29 

Requirements  Management 

27 

Project  Monitoring  and  Control 

19 

Project  Planning  and  Estimation 

35 

Scope  Management 

16 

Configuration  Management 

13 

Quality  Engineering 

20 

Risk  Control 

17 

Risk  Assessment  (With  Subcontracting  or  Without 

Subcontracting) 

20  or  19 

Total 

330-335 

Table  11.  Number  of  Questions  in  SPMEI 
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Number  of  Questions 

People  Area 

(Communication,  Teamwork,  Leadership, 

Organizational  Commitment,  Project  Manager, 

Stakeholder  Involvement,  Staffing  and  Hiring) 

164-168 

Process  Area 

(Requirements  Management,  Project  Planning  and 

Estimation,  Project  Monitoring  and  Control,  Scope 

Management) 

97 

Product  Area 

(Configuration  Management,  Quality  Engineering) 

33 

Risk  Area 

(Risk  Assessment,  Risk  Control) 

36-37 

Total 

330-335 

Table  12.  Number  of  Questions  in  SPMEI  Categorized  with  Respect  to  the  Main  Area 

2.  Basic  Design  Characteristics  of  the  Instrument 

a.  SPMEI  is  a  Self-administered  Questionnaire 

The  software  project  management  evaluation  instrument  (SPMEI)  is 
designed  to  be  used  as  a  project  management  tool  for  software  managers.  It  is  not  a 
research  instrument  for  a  specific  research  goal,  but  an  actual  project  management  tool. 
The  selection  of  a  self-administered  tool  is  driven  by  this  requirement.  SPMEI  is 
designed  in  such  a  way  that  the  software  managers  should  be  able  to  use  it  without 
difficulty.  The  pilot  studies  conducted  significantly  improved  the  wording  and  the 
usability  of  the  instrument.  During  survey  studies,  it  was  observed  that  none  of  the 
participants  had  difficulty  in  understanding  and  using  this  instrument.  In  addition,  there 

122 


was  another  significant  advantage  for  making  the  instrument  a  self-administered 
questionnaire.  This  was  an  advantage  for  data  collection  during  survey  studies.  Brace 
(2004)  states: 

Self-completion  methods,  whether  paper  based  or  electronic,  can  benefit 
from  the  complete  absence  of  an  interviewer  from  the  process.  This 
removes  a  major  source  of  potential  bias  in  the  responses,  and  makes  it 
easier  for  respondents  to  be  honest  about  sensitive  subjects. 

In  some  cases,  the  study  participants  may  feel  an  urge  to  impress  the  interviewer.  As 
Brace  pointed  out,  this  may  be  a  major  source  of  bias. 

The  reasons  for  the  selection  of  a  questionnaire-based  approach  were 
provided  previously  in  Chapter  III. 

b.  SPEMI  is  Composed  of  Sections 

This  type  of  instrument  design  is  specifically  chosen  for  two  purposes. 
First,  it  makes  the  SPMEI  a  modular  instrument.  Hence,  it  is  possible  to  replace  a  section 
with  a  better  one  in  future  studies.  In  addition,  a  section  or  a  collection  of  sections  may  be 
used  in  other  related  studies.  However,  researchers  should  be  very  careful.  Their  research 
goals  should  align  with  the  possible  uses  of  the  sections. 

Second,  it  provides  a  context  for  the  questions.  Providing  a  context  for 
statements  and  questions  decreases  the  probability  of  confusion  while  responding  to 
them.  Such  a  design  reduces  the  necessary  wording,  enabling  faster  completion  time. 

c.  SPMEI  is  User-friendly 

The  questions  in  SPMEI  are  not  open-ended  but  closed-form.  The 
respondents  are  only  supposed  to  check  boxes  where  appropriate.  Such  designs  are  user- 
friendly  by  nature  and  significantly  reduce  the  response  time  for  each  question.  Closed 
questions  include  pre-coded  responses.  Since  the  responses  are  pre-coded,  it  is  easier  to 
compare  responses.  SPMEI  is  designed  to  be  used  in  a  measurement  activity.  By  nature, 
such  activity  includes  comparison  based  on  responses  in  this  type  of  research.  A 
questionnaire  that  measures  behavior  is  likely  to  consist  of  mostly  closed  questions 
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(Brace,  2004).  Inquiring  about  the  project  on  the  identified  activities  and  entities  related 
to  project  management  may  be  considered  as  a  form  of  measuring  project  behavior.  Brace 
(2004)  states  that: 

Respondents  are  able  to  respond  relatively  easily  to  behavioral  questions, 
limited  by  only  their  memory  of  events,  the  amount  of  effort  they  are 
prepared  to  give  to  answering  the  questions  and  the  degree  to  which  they 
are  prepared  to  be  truthful. 

Thus,  closed  questions  are  preferred  in  the  design  of  the  instrument. 

d.  SPMEI  is  Comprehensive 

Software  project  management  is  complex  by  nature  (Larry  Bernstein, 
personal  communication,  August  20,  2008).  Management  of  a  software  project  involves 
many  activities  and  entities.  To  evaluate  the  project  management  effectiveness 
successfully,  it  is  imperative  to  inquire  about  project  management  of  the  project  in  many 
aspects.  Therefore,  SPMEI  had  to  be  a  comprehensive  tool.  It  includes  fifteen  project 
management  areas  and  over  330  questions.  Naturally,  the  instrument  is  not  short.  In 
research  designs,  short  and  focused  instruments  are  better.  However,  as  mentioned 
previously,  this  instrument  is  mainly  designed  to  be  a  project  management  tool  rather 
than  be  a  research  instrument. 

3.  The  Instrument  Design  Process 

An  iterative  process  was  used  in  the  design  of  the  instrument.  There  have  been  at 
least  three  major  iterations  during  the  design.  There  were  also  several  minor  iterations  to 
improve  specific  sections.  The  major  steps  in  the  instrument  design  are  listed  here  in 
order  to  guide  other  researchers  in  their  future  studies. 

a.  Step  1:  Search  for  the  Sources  of  Information 

In  this  step,  the  sources  of  information  that  can  be  used  in  the  design  of  the 
instrument  have  been  identified.  Many  sources  of  information  were  sought.  These 
sources  include  software  practitioners’  interviews,  subject  books,  related  standards,  best 
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and  worst  practice  guidelines,  journal  publications,  conference  publications,  professional 
seminars,  and  other  relevant  written  or  verbal  material  that  can  be  found  on  the  World 
Wide  Web  (WWW).  Another  source  of  information  was  personal  correspondence  with 
some  of  the  survey  study  participants.  Most  of  these  sources  are  referenced  throughout 
the  dissertation. 

This  search  for  information  was  conducted  based  on  two  main  themes. 
The  first  theme  was  software  project  management  knowledge  areas  and  practices.  Most 
of  the  relevant  information  was  found  in  the  project  management  and  software 
engineering  literature  as  well  as  via  interviews  with  practitioners.  However,  especially  for 
the  human  side  of  software  project  management,  many  sources  were  found  in  other 
disciplines  such  as  organizational  management,  sociology,  psychology,  etc.  The  second 
main  theme  was  how  to  measure  or  evaluate  the  effectiveness  of  these  project 
management  knowledge  areas  and  practices.  It  is  important  to  note  that,  especially  for  the 
guidance  of  other  researchers,  a  big  portion  of  the  infonnation  in  this  theme  did  not  come 
from  project  management  or  software  engineering  related  literature,  but  from  literature  in 
other  disciplines.  These  other  disciplines  include  sociology,  organizational  management, 
organizational  behavior,  psychology,  engineering  management,  human  resource 
management,  sales  management  and  other  related  disciplines. 

b.  Step  2:  Categorization  of  Information 

All  these  sources  identified  in  the  previous  step  were  carefully  reviewed. 
The  relevant  information  from  these  sources  was  extracted  for  use  in  the  design  of  the 
instrument.  Then  the  information  was  categorized  based  on  relevance.  A  separate  folder 
was  created  for  each  category  to  place  the  relevant  information  in  one  place. 

These  sources  were  also  rated  based  on  their  relevance  and  applicability. 
This  was  important  because  most  of  the  sources  or  studies  were  only  applicable  to  some 
extent.  In  social  sciences,  it  is  possible  to  identify  a  few  recognized  questionnaires  that 
are  commonly  used  in  studies.  For  instance,  one  such  example  is  the  organizational 
commitment  questionnaire  (OCQ)  developed  by  Porter  et  al.  (1974).  This  recognized 
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questionnaire  was  only  applicable  to  a  very  limited  extent.  However,  there  was  value  in 
reviewing  such  a  questionnaire.  The  development  of  a  few  questions  and  statements  in 
the  instrument  was  influenced  by  this  questionnaire. 

On  the  other  hand,  there  were  studies  which  influenced  the  development 
of  the  instrument  to  a  great  extent.  One  such  example  is  CMMI  vl.l  and  vl.2.  A 
significant  number  of  questions  in  some  of  the  project  management  areas  were  guided  by 
CMMI. 


c.  Step  3:  Detailed  Analysis  of  Information  Gathered 

At  the  end  of  the  first  two  steps  in  the  design  of  the  instrument,  a 
significant  amount  of  information  was  gathered.  Among  the  infonnation  gathered, 
naturally  there  was  redundancy.  This  redundancy,  to  a  certain  extent,  was  considered  as 
an  indication  of  the  importance  of  a  certain  area,  activity  or  entity.  There  were  also  pieces 
of  information  which  only  existed  in  a  few  sources.  These  were  also  carefully  reviewed 
for  inclusion  in  the  design  of  the  instrument.  Even  though  some  of  this  information  was 
referenced  in  a  few  sources  or  the  focus  of  a  few  studies,  valuable  insight  was  attained 
during  the  review.  At  the  end  of  this  step,  a  list  of  activities  and  entities  was  generated  for 
each  project  management  area.  These  lists  contained  activities  and  entities  found  related 
to  software  project  management  in  a  short,  bulleted,  categorized  form. 

d.  Step  4:  Development  of  Questions 

The  result  of  the  previous  steps  was  the  creation  of  systematic  lists  for 
each  project  management  area  in  the  3PR  framework.  For  each  item  in  these  lists,  a 
question  was  developed. 

Careful  consideration  for  the  context  was  significant  in  the  design  of  the 
instrument.  For  example,  each  section  in  the  instrument  corresponds  to  a  project 
management  area  from  the  3PR  framework.  This  provided  a  context  for  the  questions  and 
reduced  the  necessary  number  of  words  used  in  wording  the  questions. 
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The  questions  were  worded  very  carefully.  The  wording  of  the  questions 
was  kept  as  simple  and  straightforward  as  possible.  The  wording  was  very  important 
since  the  reliability  of  the  responses  was  very  much  tied  to  the  quality  of  these  questions. 

Questions  inquiring  about  similar  issues  were  closely  located  within  the 
sections  in  the  instrument.  Such  localization  reduced  the  amount  of  context  switching 
required  by  the  respondents  while  completing  the  questionnaire-based  instrument. 

The  chosen  question  types  were  closed  questions  and  statements  with  a 
Likert  scale.  These  types  enabled  faster  response  time. 

There  were  also  other  factors  considered  during  the  development  of  the 
questions.  There  are  many  specific  subject  books  focused  on  guidelines  for  the 
development  of  questionnaires.  These  books  were  consulted  extensively  whenever 
necessary.  A  good  one  on  the  topic  was  authored  by  Ian  Brace  (2004)  and  it  is  titled 
“Questionnaire  Design.” 

e.  Step  5:  Interface  Design 

The  interface  design  of  the  instrument  was  an  important  step,  thus  it  is 
specifically  mentioned  as  a  major  step.  The  first  versions  of  the  questionnaire-based 
instrument  were  more  than  sixty  pages.  This  length  is  intimidating  for  many  potential 
study  participants.  This  length  was  reduced  during  major  iterations. 

Another  important  issue  was  the  selection  of  a  specific  interface  for  each 
question.  A  number  of  different  interfaces  were  tried.  After  the  pilot  studies  on  the 
instrument,  the  latest  version  of  the  instrument  was  finalized. 

f  Step  6:  Testing  and  Redesign 

Pilot  studies  were  conducted  to  test  the  instrument.  After  these  pilot 
studies  were  concluded,  it  was  understood  that  the  content  of  the  instrument  was 
satisfactory.  Only  minor  changes  were  found  to  be  required.  On  the  other  hand,  the 
interface  was  improved  significantly.  In  the  pilot  studies,  the  study  participants  were 
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carefully  observed  for  their  reactions  to  the  instrument.  The  interface  was  an  important 
factor  that  required  special  attention  in  order  to  achieve  better  results.  The  interface  took 
its  final  form  after  the  pilot  studies. 

4.  Question  Types  in  SPMEI 

In  SPMEI,  there  are  three  types  of  questions.  In  the  first  type,  the  respondent  is 
requested  to  select  the  statement  or  statements  that  apply  to  the  project.  This  question 
type  is  extensively  used  in  the  design  of  SPMEI.  An  example  of  this  type  of  question  is 
provided  below. 

RC6.  Check  the  statement/s  that  applies  to  the  project.  (Check  only  one.) 

i'H  Adequate  slack  time  is  planned  in  the  schedule  for  consequences  due  to  risks. 

[  I  There  is  not  any  slack  time  planned  for  consequences  due  to  risks. 

O  Not  enough  slack  time  is  planned  in  the  schedule  for  consequences  due  to  risks. 

In  the  second  question  type,  the  respondent  is  asked  about  an  aspect  of  the  project 

management.  A  question  taken  from  the  communication  section  of  the  instrument  is 

presented  below. 

C2.  Who  are  generally  present  in  the  project  status  meetings?  (Check  all  that 
apply.) 

PI  Project  manager 
I  Project  team  leaders 
!  |  Project  team  members 
1  |  Customer/s  and/or  user  representatives 
|  |  Various  stakeholders  or  stakeholder  representatives 
I  I  Executive  management  /  Project  sponsor 


The  third  type  of  question  uses  a  statement  associated  with  a  Likert  scale,  which 
is  a  psychometric  scale  that  is  commonly  used  in  questionnaires.  The  Likert  scale  is 
frequently  known  as  an  “agree-disagree”  scale  (Brace,  2004).  This  technique  is  easy  to 
distribute  in  self-administered  questionnaires.  Often,  responses  using  the  Likert  scale  are 
associated  with  scores.  These  scores  may  be  from  1  to  5,  negative  or  positive,  or  -2  to  +2 
(Brace,  2004).  Brace  states  that  “as  these  are  interval  data,  means  and  standard  deviations 
can  be  calculated  for  each  statement.”  An  example  to  this  type  of  question  is  presented 
below. 
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Automated  requirements 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

RM9 

development  and  management 
tools  are  used. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

Project  management  efforts  naturally  employ  a  set  of  best,  worst  and  common 
practices.  Jones  (2004)  analyzed  about  250  large  software  projects  for  software  project 
management  practices.  In  his  analysis,  he  identified  a  set  of  factors  associated  with 
successful  and  failed  projects.  One  of  the  factors  is  change  control  management.  He 
identified  that  while  effective  change  control  management  is  a  factor  in  achieving 
success,  ineffective  change  control  management  is  a  factor  in  failure.  In  this  particular 
example,  effective  change  control  management  may  be  considered  as  an  example  for  a 
best  practice  associated  with  success.  Ineffective  change  control  management  may  be 
considered  as  an  example  for  a  worst  practice  associated  with  failure.  For  example, 
SPMEI  investigates  the  project  for  the  existence  and  quality  of  change  control 
management  related  practices  in  its  various  sections  such  as  scope  management  and 
requirements  management.  Conducting  project  status  meetings  may  be  considered  as  an 
example  of  common  practice.  Today,  in  most  projects,  project  status  meetings  are  held 
with  the  participation  of  various  people  at  various  times.  With  broad  involvement  of 
necessary  stakeholders,  the  items  discussed  in  these  meetings  detennine  the  effectiveness 
or  the  quality  of  project  status  meetings.  SPMEI  also  investigates  such  practices. 

Project  management  best,  worst  and  common  practices  result  in  a  set  of  activities 
and  entities.  SPMEI  investigates  the  effectiveness  of  activities  and  entities  related  to 
project  management  in  four  different  approaches.  Examples  will  be  provided  for  each 
approach. 


a.  Approach  1:  The  Existence  of  an  Activity 

In  this  approach,  the  existence  of  a  certain  activity  is  sought.  This  activity 
is  generally  the  result  of  a  best  practice.  The  example  below  is  taken  from  the 
configuration  management  section  of  SPMEI. 
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CM3.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

I  |  Baselines  and  configuration  items  are  identified  at  the  beginning  of  the  project 
and  updated  as  necessary. 

U  The  owner  or  responsible  staff  is  identified  for  each  configuration  item. 

I  |  Every  configuration  item  has  a  unique  identifier. 

I  |  Important  characteristics  for  each  configuration  item  are  identified  such  as 
author,  type,  date,  version  number  etc. 
j  |  None 

In  the  second  statement  above,  SPMEI  gathers  project  data  whether  the 
owner  or  responsible  staff  is  identified  for  each  configuration  item  or  not.  This  particular 
activity  is  a  practice  from  Capability  Maturity  Model  Integration  1.1  (CMMI  vl.l).  This 
practice  is  listed  as  a  subpractice  under  the  identify  configuration  items  specific  practice 
of  configuration  management  process  area  in  CMMI  vl.l  (CMMI  vl.l  Continuous 
Presentation,  page  504). 

b.  Approach  2:  The  Existence  of  an  Entity 

In  this  approach,  the  existence  of  a  certain  entity  is  sought.  During  project 
development,  the  best  practices  result  in  certain  entities.  For  example,  an  effective 
configuration  management  requires  the  development  of  configuration  management 
document,  the  establishment  of  a  configuration  control  board,  and  generation  of  a 
configuration  item  list.  SPMEI  searches  the  existence  of  these  entities  as  follows  in  the 
example  below. 

CM2.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

1  |  There  is  a  configuration  management  document. 

There  is  a  configuration  or  change  control  board,  committee  or  team. 
j  I  There  is  a  configuration  items  list. 

I  I  None 


c.  Approach  3:  How  Well  an  Activity  is  Conducted 

In  this  approach,  SPMEI  gathers  data  on  the  rigor  or  the  quality  of  certain 
activities.  Jones  (1998)  emphasizes  the  importance  of  automation  in  project  management 
by  stating,  “...the  lagging  projects  tend  to  be  essentially  manual  for  most  project 
management  functions.  The  leading  projects  deploy  a  notable  quantity  of  quality  control 
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and  project  management  automation.'"  In  the  example  below,  the  rigor  in  using  the 


automating  of  project  management  tools  in  planning  the  project  is  inquired. 


PPE 

22 

Various  project  management 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

tools  are  used  in  planning  the 
project. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

d.  Approach  4:  The  Rigor  or  the  Quality  in  the  Existence  of  an 
Entity 


In  this  final  approach,  SPMEI  gathers  data  on  the  rigor  or  the  quality  of 


certain  entities.  Having  more  experienced  project  team  members  than  inexperienced 
project  team  members  is  an  obvious  advantage  for  project  organizations.  This  aspect  is 
inquired  in  SPMEI  as  follows  in  the  example  below. 


There  are  more  experienced 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

T14 

team  members  than  the 

Agree 

Disagree 

Applicable 

inexperienced  team  members. 

□ 

□ 

□ 

□ 

□ 

□ 

5.  Optional  Questions  in  SPMEI 

In  SPMEI,  some  of  the  questions  are  only  applicable  when  certain  conditions 
exist  in  the  project.  These  questions  and  the  conditions  are  presented  in  Table  13. 


Question  Identifier 

Condition 

L17 

When  the  team  mostly  consists  of  inexperienced  staff 

L18 

When  the  team  mostly  consists  of  experienced  staff 

SI11-SI12 

When  the  project  is  developed  for  the  market  without 

a  specific  contract 

SI 13-SI 1 8 

When  the  project  is  developed  under  a  contract  with  a 

customer 
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RA20 


When  subcontracting  is  utilized  during  the  project 
Table  13.  Optional  Questions  in  SPMEI  Based  on  Certain  Conditions 

6.  Other  Significant  Characteristics  of  the  Instrument 

Other  notable  characteristics  of  the  SPMEI  are  highlighted  in  the  sections  below. 

The  instrument  is  only  applicable  to  the  projects  that  have  conducted  within 
a  certain  period.  Many  scholars  would  agree  that  the  concepts  of  projects  and 
management  of  projects  date  back  to  the  early  days  of  civilization.  “Projects  have  been 
the  part  of  human  scene  since  civilization  started”  (Lock,  1987). 

Managing  projects  is  one  of  the  oldest  and  most  respected 
accomplishments  of  mankind.  We  stand  in  awe  of  the  achievements  of  the 
builders  of  the  pyramids,  the  architects  of  cities,  the  masons  and  craftsmen 
of  great  cathedrals  and  mosques;  of  the  might  and  labor  behind  the  Great 
Wall  of  China  and  other  wonders  of  the  world  (Morris,  1994). 

Some  of  the  principles,  activities  and  concepts  that  are  used  in  those  early  days  of 
the  civilization  exist  today  even  though  the  application  of  them  may  have  changed.  For 
example,  Cooke-Davies  (2001)  states,  “the  subdivision  of  manpower  into  smaller  units 
for  the  purposes  of  oversight  appears  to  have  been  well  established  in  the  ancient  world. 
The  first  recorded  reference  to  a  supervisor  dates  from  1750  B.C.”  The  idea  of 
subcontracting  again  dates  back  to  the  early  days,  such  as  the  Colosseum  being  built  by 
four  contractors  (Morris,  1994).  “Modern  project  management  is  built  on  foundations 
nearly  as  old  as  civilization  itself’  (Cooke-Davies,  2001). 

Cooke-Davies  divides  the  history  of  project  management  into  four  eras: 

1.  Projects  in  a  pre-  and  proto-capitalist  society  (before  1850). 

2.  The  era  of  classic  capitalism:  project  management  from  1850  to  1950. 

3.  The  era  of  “managerial  capitalism”:  project  management  from  1950  to  the 
mid-1980s. 

4.  The  era  of  “intellectual  capitalism”:  project  management  since  the  mid- 
1980s. 
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In  each  of  these  eras,  a  certain  social  environment  and  emergent  concepts 
dominate  that  era.  Table  14  provides  these  eras  in  a  tabular  format. 


Social  Environment 

Emergent  concepts 

I .  Projects  in  a  pre-  and  proto¬ 
capitalist  society,  (before  c.  1 850). 

a)  Mobilisation  and  management 
hierarchy. 

b)  Client-contractor  relationship 

2.  Projects  and  project 
management  in  a  world  dominated 
economically  by  "classic 
capitalism",  (c.  1 850  to  c.  1 950) 
Evolution  of  formal  techniques. 

c)  Management  disciplines 

d)  Project  planning  techniques. 

e)  Logistics  planning. 

f)  Work  breakdown  structure. 

g)  Time-driven  research. 

3.  Projects  and  project 
management  during  the  era  of 
"managerial  capitalism”,  (c.  1 950  to 
mid-1980s) 

Birth  of  a  profession. 

Widespread  adoption  of  project 
management  by  the  engineering  and 
construction  industries  in  the  1970s. 
Early  application  of  project 
management  techniques  in  other 
industries  such  as  IT  and 
Entertainment. 

The  expansion  of  project 
management  from  "traditional” 
environments  into  strategic  business 
management  and  IT  during  the 

1980s. 

h)  Systems  management. 

i)  Scheduling  techniques. 

j)  Procurement  management. 

k)  Performance  measurement 
(Earned  Value  Analysis). 

l)  Programme  management. 

m)  Project  management 
professional  societies. 

n)  Project  matrix  organisations. 

o)  Refinement  of  defence 
development  life-cycles. 

p)  Widespread  acceptance  of 
project  management  practices. 

q)  New  forms  of  contract  -  BOO 
(T). 

r)  Macro-technology  programmes. 

s)  Application  of  project 
management  to  IT  projects. 

t)  Systems  engineering  and 
software  project  management. 

4.  Projects  and  project  management 
during  an  age  of  “intellectual 
capitalism”,  (mid-1980s  and 
beyond). 

Project  management  as  a  "core 
business  discipline": 

The  current  trend  towards  the 
“project-based  organisation”  and 
“management  by  projects”. 

u)  Business  process  re-engineering. 

v)  Project-based  organisations. 

w)  New  contract  philosophies 
(Partnering,  Alliancing). 

x)  Attempts  to  apply 
“Benchmarking”  techniques  to 
projects 

Table  14.  Origin  of  Elements  Present  in  Current  Project  Management  Practice  [From 

Cooke-Davies,  2001)] 

In  the  era  of  managerial  capitalism  from  1950  to  mid  1980s,  systems  engineering 
and  software  project  management  are  among  the  emergent  concepts.  During  this  era, 
there  were  significant  advancements  in  the  computing  as  well  as  project  management 
fields.  For  example,  1969  is  the  year  that  the  Project  Management  Institute  (PMI)  was 
formed.  In  1981,  PMI  started  the  effort  for  the  first  edition  of  “A  Guide  to  the  Project 
Management  Body  of  Knowledge,”  (PMBOK).  This  is  the  result  of  building  a  knowledge 
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base  in  this  era.  Again  in  this  era,  the  number  of  software-based  system  projects 
exponentially  increased.  Towards  the  end  of  this  era,  the  experiences  gained  from 
managing  software  projects  were  to  take  their  place  in  the  project  management  literature. 
A  good  example  is  the  work  “The  Mythical  Man-Month”  by  Frederick  Brooks  in  1975 
(Brooks,  1975).  Another  example  is  the  work  “Software  Engineering  Economics”  by 
Barry  Boehm  in  1981  (Boehm,  1981).  The  principles  stated  in  these  and  other  similar 
works  by  various  scholars  guided  many  software  projects  in  the  era  of  intellectual 
capitalism.  In  the  last  era  of  project  management,  management  of  software  projects  has 
become  more  systematic. 

The  development  of  the  SPMEI  was  implicitly  guided  by  two  sets  of  principles. 
The  first  set  of  these  may  be  considered  as  time-independent  principles.  This  is  because 
these  are  the  principles  that  have  existed  since  the  early  days  of  civilization  and  are  still 
applicable  today.  Naturally,  these  principles  guided  the  development  of  a  certain  portion 
of  questions  in  the  instrument.  The  second  set  of  principles  may  be  considered  as  time- 
dependent  principles.  These  principles  are  derived  by  the  needs  of  the  current  social 
environment.  Therefore,  their  applicability  is  limited  within  a  specific  period.  A  big 
portion  of  the  instrument  is  developed  by  the  guidance  of  these  time-dependent 
principles.  As  a  result,  the  SPMEI  is  only  applicable  to  those  projects  that  are  conducted 
in  the  last  era  of  project  management,  which  is  described  as  intellectual  capitalism  by 
Terence  J.  Cooke  Davies. 

The  author  has  conducted  two  test  cases  that  support  the  argument.  One  of  the  test 
cases  is  conducted  on  a  software  project  that  took  place  in  1974.  The  test  case  shows  that 
the  SPMEI  and  the  evaluation  model  are  not  applicable  to  this  project.  The  other  test  case 
is  conducted  on  a  software  project  that  took  place  in  1984.  This  test  case  shows  that  the 
instrument  and  the  evaluation  model  are  applicable  to  this  project.  Therefore,  it  is 
possible  to  assume  that  the  SPMEI  and  the  evaluation  model  are  applicable  to  the  projects 
that  were  conducted  after  the  1970s.  Another  question  that  is  of  interest  is  the  time  the 
instrument  becomes  inapplicable.  This  is  a  hard  question  to  answer  since  it  requires  a 
good  prediction  of  future  advancements  in  the  software  project  management  field.  Unless 
there  are  breakthroughs,  with  the  observed  rate  of  advancements  in  software  engineering 
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and  project  management  fields,  it  is  possible  to  assume  the  SPMEI  and  the  evaluation 
model  will  be  applicable  to  software  projects  that  will  be  conducted  in  the  next  15-20 
years.  This  is  based  on  the  past  progression  in  the  knowledge  base  of  project  management 
in  the  era  of  intellectual  capitalism. 

The  existence  of  these  two  sets  of  principles  guiding  the  development  of  the 
instrument  has  two  implications.  First,  because  of  the  time-independent  principles,  a 
certain  portion  of  the  SPMEI  may  be  reused  by  researchers  in  the  future.  Therefore,  the 
existence  of  SPMEI  saves  time  and  effort  for  future  research  works.  Second,  because  of 
the  time-dependent  principles,  the  use  of  the  instrument  and  the  evaluation  model  are 
applicable  to  projects  that  are  conducted  within  a  certain  period. 

The  instrument  is  only  applicable  to  software  or  software  intensive 
development  projects.  It  is  not  applicable  to  software  maintenance  projects.  In  the 

life  cycle  of  a  project,  the  maintenance  phase  of  the  project  is  significant  in  many  ways. 
According  to  Schach  (2002),  67%  of  the  project  total  cost  is  devoted  to  the  maintenance. 
The  maintenance  phase  generally  starts  when  the  project  deliverables  (the  products,  the 
services,  the  manuals,  etc.)  are  handed  to  the  customer.  After  this  milestone,  all  activities 
related  to  the  changes  in  the  deliverables  are  considered  as  maintenance  activities.  Prior 
to  this  milestone,  all  activities  related  to  the  project  are  considered  as  development 
activities.  This  milestone  has  important  significance  for  the  purposes  of  this  research  as 
well  as  other  purposes.  This  milestone  in  the  project  life  cycle  is  the  cornerstone  for  many 
changes  in  the  context  or  environment  of  the  project.  The  first  context  change  is  that  the 
project  deliverables  are  no  longer  being  developed  by  the  project  team  but  they  are  in 
operational  use  by  the  users  and  the  customers.  The  second  important  context  change  is 
that  the  development  team  dissolves  in  many  cases.  The  project  manager,  the  project 
team  leaders  and  many  other  stakeholders  move  on  to  other  projects.  Another 
significance  of  this  milestone  is  that  the  project  budget  is  estimated  based  on  this 
milestone  in  most  cases.  In  some  cases,  the  project  is  even  handed  to  the  customer  when 
the  project  funding  runs  out.  Testing  is  the  last  phase  of  the  software  development  effort. 
If  the  activities  before  testing  cost  more  than  expected,  the  amount  of  testing  gets  cut  to 
meet  the  budget.  The  same  treatment  is  also  true  for  the  project  schedule.  Basically,  the 
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project  variables  that  drive  the  project  plans  are  all  based  on  this  milestone  in  the  life 
cycle.  Even  though  the  maintenance  phase  of  the  project  in  the  life  cycle  of  a  project  is 
widely  accepted  as  a  natural  extension,  for  the  project  planning  and  estimation  purposes 
this  milestone  is  considered  the  end  of  the  project.  For  many  years,  project  success  was 
evaluated  based  on  delivering  the  project  on  time,  within  budget  and  with  expected 
functionality.  All  these  variables  relate  to  this  milestone.  For  the  sake  of  raising  an 
argument,  isn’t  that  contradictory  when  the  project  maintenance  is  considered  a  natural 
phase  in  the  software  evolution,  when  all  the  project  planning  and  estimation  is  targeted 
to  the  end  of  development  phase?  Another  interesting  observation  is  that  PMI’s  PMBOK 
(2004)  does  not  include  a  section  for  maintenance  phase. 

All  these  context  changes  naturally  affect  the  project  management  principles 
deriving  the  activities  in  the  project  life  cycle.  Even  though  there  are  many  studies  on  the 
technical  aspects  of  the  maintenance  phase,  the  literature  lacks  studies  on  managing  the 
maintenance  phase  of  projects.  It  is  the  author’s  belief  that  managing  the  maintenance 
activities  may  rely  on  different  project  management  principles  than  project  development 
activities.  For  example,  most  current  project  estimation  methods  and  approaches  are 
based  on  estimating  initial  development  activities.  There  is  a  set  of  activities  called 
reverse  engineering  that  come  into  play  during  maintenance  of  legacy  code.  Reverse 
engineering  activities  are  different  then  development  activities.  Management  of  the 
maintenance  phase  seems  a  prospective  area  for  future  research. 

There  are  three  types  of  maintenance  (Schach,  2002).  They  are  corrective, 
perfective,  and  adaptive  maintenance.  It  is  possible  to  argue  that  all  these  maintenance 
activities  may  also  be  considered  another  project  by  themselves.  Some  even  make  the 
distinction  between  development  and  maintenance  activities  by  dividing  software 
projects  into  two  categories  such  as  software  development  projects  and  software 
maintenance  projects.  Stating  this  distinction  is  a  sign  that  there  is  a  difference  between 
software  development  and  maintenance  projects. 

Whether  the  maintenance  activities  are  considered  a  separate  but  related  project  or 

a  natural  extension  of  the  project  development  activities,  managing  these  sets  of  activities 

is  different  then  managing  development  activities.  Munns  and  Bjeirmi  (1996)  and  Cooke- 
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Davies  (2001)  argue  that  the  scope  of  project  management  success  is  up  until  to  the 
handover  (see  Figure  36).  The  test  cases  presented  in  the  following  sections  align  with 
the  argument.  Since  the  instrument  is  focused  on  software  development  projects,  it  is 
therefore  not  applicable  to  software  maintenance  projects.  SPMEI  scope  excludes  the 
maintenance  phase  in  the  life  cycle  of  a  project. 
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Figure  36.  The  Scope  of  Success  within  Project  Life  Cycle  [From  (Munns  and  Bjeirmi, 

1996)] 


The  scope  of  the  instrument  is  limited  to  the  project  phases  between 
conception  and  delivery  (handover).  Figure  36  above  already  depicts  the  scope  of 
project  management  success.  Munns  and  Bjeirmi  (1996)  explain  in  detail  what  the  scope 
of  project  success  and  project  management  successes  are  and  why.  De  Wit  (1988)  makes 
a  distinction  between  project  success  and  project  management  success.  Furthermore, 
Cooke-Davies  (2004a)  lists  three  levels  of  success: 

1 .  Project  Management  Success:  Was  the  project  done  right? 

2.  Project  Success:  Was  the  right  project  done? 

3.  Consistent  Project  Success:  Were  the  right  projects  done,  right  time  after 
time? 

Detailed  discussions  regarding  the  different  scopes  were  presented  earlier.  Why 
maintenance  projects  or  the  maintenance  phase  of  the  projects  is  out  of  the  scope  of  the 
instrument  is  presented  in  the  previous  section. 
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According  to  Munns  and  Bjeirmi  (1996),  the  conception  phase  of  a  project  is 
when  the  idea  for  the  project  is  birthed  within  the  client  organization  and  its  feasibility 
determined.  Basically,  in  this  phase  the  decision  to  undertake  or  not  to  undertake  the 
project  is  determined.  This  decision  is  driven  by  many  internal  and  external  factors.  Some 
of  these  factors  are  the  problem  to  be  solved,  the  applicability  of  the  implementation 
alternatives,  aligning  and  resolving  the  conflicting  concerns  of  all  stakeholders,  the 
adequacy  of  resources,  the  availability  of  project  personnel  and  the  skills  needed  to 
successfully  complete  the  project,  the  changing  market  dynamics,  the  availability  of  the 
necessary  technologies,  competing  organizations,  other  similar  and  supplementary 
products  in  the  market,  the  social  and  political  environment,  etc.  This  list  is  not  complete 
by  any  means.  Most  definitions  of  a  project  state  that  the  project  is  unique.  Therefore,  for 
every  project,  the  factors  may  be  quite  different,  and  these  factors  influence  the  project 
“go”  decision.  Some  scholars  and  practitioners  argue  that  the  success  of  the  project  is 
determined  in  this  phase  even  before  any  implementation  takes  place.  Boehm  and  Jain 
(2005)  state  that: 

...software-intensive  enterprises  and  their  success  are  subject  to  multiple 
concurrent  influences,  some  of  which  are  unpredictable.  For  example,  a 
project  that  is  poorly  requirements-engineered  and  architected,  poorly 
managed,  behind  schedule,  and  over  budget  can  still  turn  into  a  great 
success  with  the  appearance  of  just  the  right  new  COTS  product  to  satisfy 
stakeholder  needs.  The  reverse  is  true  as  well. 

Cooke-Davies  states  that  different  stakeholders  may  have  different  success 
criteria.  Such  differences  make  the  measurement  of  project  success  a  complex  and 
inexact  matter.  What  is  considered  a  success  for  one  stakeholder  may  be  considered  as  a 
disaster  for  another  stakeholder.  What  can  appear  a  success  one  day  may  be  a  failure  the 
next  day  (Cooke-Davies,  2001).  Other  prominent  scholars  in  the  project  management 
area,  such  as  Pinto  and  Slevin,  stress  the  difficulties  in  establishing  what  constitutes  a 
success. 

The  scope  of  this  instrument  is  the  same  as  the  views  of  Munns  and  Bjeirmi 
(1996)  as  depicted  in  the  Figure  36.  The  focus  of  the  instrument  is  aligned  with  the  views 
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of  Cooke-Davies  for  project  management  success:  “Was  the  project  done  right?”  As  a 
result,  the  instrument  inquires  how  well  the  project  is  managed. 

SPMEI  is  a  self-evaluation  instrument.  Software  project  management  is 
complex  by  nature  and  has  many  aspects.  Not  every  stakeholder  is  subject  to  the  “big 
picture”  of  the  project.  The  software  project  management  evaluation  instrument,  SPMEI, 
should  only  be  used  by  a  person  who  has  extensive  knowledge  and  understanding  of 
various  aspects  during  project  development.  Generally,  this  person  is  the  project  manager 
or  the  technical  manager  of  the  project.  This  person  may  also  be  an  executive  manager,  a 
project  team  leader,  a  project  metrics  or  process  engineering  expert,  or  even  a  developer 
when  the  project  team  is  not  big.  SPMEI  is  not  designed  to  be  used  by  an  outsider  such  as 
a  researcher  or  a  stakeholder  who  is  only  subject  to  certain  limited  aspects  of  the  project. 
The  self-evaluation  characteristic  of  the  SPMEI  is  the  result  of  this  natural  requirement. 
The  instrument  is  designed  based  on  this  assumption.  It  involves  inquiries  requiring  a 
response  from  a  person  who  has  first-hand  experience  in  the  complex  dynamics  of  the 
project  management. 

Such  a  characteristic  raises  an  important  issue.  It  is  possible  that  this  person  may 
not  be  objective  in  responding  to  the  questions  in  the  instrument;  therefore  the  resulting 
metric  may  not  be  reliable.  Even  though  this  is  a  valid  issue,  in  practice  the  occurrences 
of  such  evaluations  will  not  be  common.  First  of  all,  the  instrument  is  designed  in  such  a 
way  that  strong  biases  are  likely  to  be  identified.  The  instrument  is  extensive  and 
inclusive.  It  includes  fifteen  project  management  areas.  Because  of  the  nature  of  project 
management,  the  areas  are  closely  tied  to  each  other.  For  example,  an  effective  risk 
control  can  only  be  the  result  of  an  effective  risk  assessment.  Effective  teamwork  can  be 
achieved  via  effective  communication,  an  able  project  manager,  effective  leadership  of 
various  leaders  in  the  project  organization  and  commitment  from  stakeholders.  Therefore, 
inconsistencies  among  responses  in  related  areas  will  reveal  intentional  and  unintentional 
biases.  Second,  the  instrument  will  likely  be  used  by  managers  and  organizations  that  are 
committed  to  achieve  better  results  in  projects.  These  managers  and  organizations  value 
candid  assessments  and  improvement  efforts.  Thus,  there  is  no  value  for  these  managers 
and  organizations  when  evaluations  are  not  based  on  candid  responses.  The  expectance  is 
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that  the  instrument  will  likely  to  be  used  in  this  context.  There  is  a  solution  to  this  issue. 
The  instrument  may  be  applied  by  two  individuals  satisfying  the  condition  that  these 
individuals  are  one  of  the  stakeholders  mentioned  above.  For  example,  a  project  manager 
and  a  team  leader  may  apply  the  instrument  at  the  same  time,  and  evaluations  based  on 
the  responses  of  these  individuals  are  compared  to  identify  when  the  existence  of  a  bias  is 
suspected. 

The  goal  of  the  project  management  effectiveness  evaluation  is  to  provide 
feedback  to  the  interested  stakeholders.  In  most  cases,  the  customers  or  the  end  users 
would  not  be  interested  in  the  quality  of  the  project  management  as  long  as  the  project 
result,  the  product  or  the  service,  satisfies  their  need.  A  good  question  would  be:  why 
would  the  project  managers  need  a  tool  such  as  SPMEI  to  evaluate  the  project 
management  effectiveness  or  to  identify  the  project  problems  when  they  already  have  an 
intuition  based  on  experience?  Don’t  they  already  know  what  the  project  problems  are? 
In  most  cases,  team  members  including  project  managers  have  a  sense  of  the  project 
management  quality.  They  may  not  need  a  tool  to  say  what  they  may  already  know. 
However,  they  need  SPMEI  and  the  evaluation  model  when  it  is  time  to  convince  other 
stakeholders  for  the  reasons  of  ineffectiveness  during  project  development.  Because  most 
project  management  related  problems  require  a  solution  that  includes  commitments  from 
a  range  of  project  stakeholders.  A  few  examples  of  these  problems  are  inadequate 
funding,  lack  of  commitment  from  the  users  for  necessary  participation  in  requirements 
development  or  testing  phase,  the  need  for  a  more  realistic  schedule,  the  need  to  stabilize 
the  requirements,  etc.  The  evaluations  based  on  SPMEI  and  the  accompanying  evaluation 
model  will  empower  the  software  managers  with  a  scientific  systematic  tool  that  will  help 
them  to  convince  other  project  stakeholders  for  more  commitment.  The  analysis  of  two 
survey  studies  conducted  for  this  research  revealed  that  even  though  the  technical 
managers  objected  to  the  project  schedule  due  to  infeasibility  at  the  beginning  of  the 
project,  the  projects  went  ahead  with  their  current  schedule  estimations.  Also,  in  both  of 
these  projects  the  customer  and  the  users  did  not  participate  enough  to  make  those 
schedules  possible.  In  one  of  the  cases,  the  project  suffered  from  schedule  slip.  In  the 
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other,  the  project  delivered  much  less  functionality  than  planned.  SPMEI  helps  the 
managers  to  make  their  cases  to  other  stakeholders.  It  is  possible  to  prevent  such 
problems  with  the  use  of  a  PME  metric. 

In  other  cases,  it  is  possible  that  software  managers  lose  the  big  picture  and  have  a 
tainted  view  regarding  the  quality  of  the  project  management.  SPMEI  provides  and 
reminds  them  of  the  big  picture  in  these  cases.  It  is  possible  for  software  managers  to  be 
carried  away  with  the  day-to-day  problems  of  the  project  and  they  may  lose  focus. 
Organizational  politics  is  a  prime  source  of  such  problems.  If  used  as  a  monitoring  tool, 
SPMEI  will  help  software  managers  to  focus  on  the  big  picture  of  the  project. 

SPMEI  may  also  be  used  as  a  monitoring  tool.  The  primary  goal  of  the 
instrument  is  to  be  used  as  an  evaluation  tool  after  the  project  is  completed.  The  feedback 
from  the  evaluation  is  to  be  used  as  guidance  for  the  upcoming  projects.  The  instrument 
and  the  evaluation  model  provide  the  best  evaluation  when  the  project  is  delivered  to  the 
customer.  During  the  development  of  the  instrument,  it  became  clear  that  the  tool  can 
also  be  used  as  a  project  monitoring  tool.  To  confirm  such  a  hypothesis,  one  of  the 
surveys  was  conducted  on  a  project  that  was  in  its  implementation  phase.  The  study  result 
supported  the  hypothesis. 

The  earliest  that  the  evaluations  may  be  perfonned  is  after  some  requirements 
development  activities  are  carried  out  because  by  the  time  the  project  reached  this  point, 
many  of  the  essential  project  management  related  activities  are  already  conducted.  These 
activities  include: 

The  project  manager  is  already  chosen.  The  project  manager’s  role  is 
shaped  with  the  influence  of  many  factors  including  political  and  social 
factors. 

The  project  is  staffed  to  a  certain  threshold.  The  project  team  structure 
becomes  clear.  The  project  organization  is  identified. 

Ideally,  all  stakeholders  should  have  been  identified  by  this  time.  Even  if 
not,  the  concerns  of  primary  stakeholders  and  the  conflicting  agendas  start 
affecting  the  project  in  many  ways. 

Most  planning  and  estimation  activities  are  carried  out. 
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Hopefully,  the  project  scope  is  clear  at  this  point.  In  the  case  that  the  scope 
is  not  clear,  naturally  there  is  an  effort  to  identify  the  proper  scope. 

Many  supplementary  systems  should  be  in  place.  For  example,  necessary 
communications  systems,  configuration  management  systems,  project 
databases  and  other  automated  systems  required  for  the  project  execution. 
Ideally,  the  quality  policy  should  be  clear.  Even  though  there  is  not  an 
explicit  quality  policy,  the  project  team  already  has  an  idea  of  what  the 
overall  quality  will  be  in  the  project. 

Project  monitoring  and  control  procedures  should  be  in  place. 

Project  communication  procedures  should  be  in  place. 

The  work  breakdown  structure  or  similar  document  has  at  least  its  overall 
structure. 

Project  risks  should  have  been  identified.  Risk  management  procedures 
should  be  in  place. 

Requirements  management  procedures  should  be  in  place. 

The  list  above  is  only  a  collection  of  activities  that  should  have  been  conducted  by 
the  time  some  of  the  requirements  development  activities  are  carried  out.  At  this  point  in 
the  life  cycle  of  the  project,  political  issues  are  already  pulling  the  project  to  a  certain 
course.  Requirements  development  is  the  phase  in  which  the  effects  of  these  forces  are 
reflected  in  the  requirements  documents.  Project  politics  are  being  shaped.  Executive 
management  desires  to  maximize  their  profit  by  either  providing  less  functionality  while 
abiding  to  the  contract  or  pushing  the  project  team  to  work  overtime.  The  customers  and 
users  desire  all  the  functionality  and  perfonnance  they  can  get  out  of  the  project  while 
keeping  the  cost  low.  The  project  team  desires  to  provide  a  high  quality  product  that  they 
can  be  proud  of.  Secondary  stakeholders  want  their  issues  resolved  and  their  concerns 
reflected  in  the  requirements  document.  Some  scholars  pointed  out  that  to  a  high  degree 
project  success  is  determined  by  the  decisions  in  the  project  planning  phase.  For  example, 
O’Connell  (2002)  states  that  the  fate  of  the  project  is  sealed  in  the  planning  phase.  In  his 
view,  the  planning  phase  includes  requirements  development.  A  prominent  project 
management  scholar  Jeffrey  K.  Pinto  stresses  that: 


142 


In  my  research  and  consulting  experience,  I  have  found  that  most 
companies  spend  thousands  of  hours  planning  and  implementing  a 
multimillion  or  even  multibillion  dollar  investment,  developing  intricate 
plans  and  schedules,  fonning  a  cohesive  team,  and  maintaining  realistic 
specification  and  time  targets,  all  to  have  the  project  derailed  by  political 
processes.  This  is  a  pity,  particularly  because  the  end  result  is  often 
foreseeable  early  in  the  development  of  the  project-  usually  as  the  result  of 
a  project  manager’s  refusal  to  acknowledge  and  cultivate  political  ties, 
both  internal  to  the  organization  and  externally  with  the  clients  (Pinto  and 
Slevin,  1998,  p.  257). 

As  Pinto  stresses,  the  project’s  result  is  often  foreseeable  early  in  the  development 
of  the  project.  SPMEI’s  capability  of  being  a  project-monitoring  tool  is  no  coincidence.  It 
is  possible  to  argue  that  such  capability  is  driven  by  project  reality.  By  the  time  the 
project  is  in  the  requirements  development  phase,  most  planning  activities  are  conducted 
and  project  dynamics  reach  a  certain  threshold  with  considerable  influence  from  project 
politics.  Therefore,  after  this  point  in  the  life  cycle  of  the  software  project,  SPMEI-based 
project  management  effectiveness  evaluations  may  be  conducted  in  a  periodic  or 
aperiodic  manner.  Such  use  of  the  tool  enables  project  managers  to  monitor  their  projects. 

Why  does  the  SPMEI  include  fifteen  project  management  areas?  Why  is  it 
not  fourteen  or  sixteen?  In  this  study,  the  total  number  of  areas  in  SPMEI  is  only  driven 
by  the  research.  In  the  beginning  of  the  study,  there  was  not  a  target  number  for  the  total 
number  of  areas  that  should  be  included  in  the  design  of  SPMEI.  The  3PR  framework 
guided  the  design  of  SPMEI  and  the  areas  within  it.  Project  management  is  a  complex 
endeavor.  The  study  conducted  by  Fortune  and  White  (2006)  listed  twenty-six  different 
critical  success  factors  extracted  from  the  review  of  sixty-three  publications. 
Furthermore,  not  all  studies  include  the  same  factors.  Such  study  indicates  the  complexity 
involved  in  realizing  a  successful  project.  A  quote  from  the  personal  communications 
with  Dr.  Larry  Bernstein  (2008)  is,  “...real  project  management  does  not  lead  to  simple 
answers.  Things  are  by  their  nature  complex  and  need  analysis.” 

Therefore,  the  instrument  concentrates  on  not  just  a  few  areas  but  a  broad  number 
of  areas.  The  more  questions  we  ask  regarding  the  project  management,  the  more  insight 
we  will  have  on  the  success  or  failure  of  the  projects,  assuming  the  questions  are  based 
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on  a  sound  research  design  and  the  responses  are  candid.  In  the  survey  study  conducted, 
there  were  seventeen  areas  inquired  via  question  #5  (Appendix  D).  The  areas  of  support 
activities  and  technical  complexity  were  excluded  from  the  design  of  SPMEI.  The  results 
of  the  survey  indicate  that  these  areas  are  the  least  important  areas  among  those  listed. 
These  areas  are  excluded  in  the  design  of  the  SPMEI  mainly  to  limit  the  length  of  the 
instrument.  Even  though  we  would  like  to  get  more  data  by  asking  more  questions,  we 
may  have  to  limit  the  length  of  the  overall  procedure.  If  the  data  gathering  procedure 
takes  too  long,  the  study  participants  may  get  tired  toward  the  end  of  the  procedure.  Thus, 
the  reliability  of  the  responses  may  be  negatively  affected  due  to  this.  A  balance  in  the 
design  of  experimentation  between  these  two  factors  should  be  sought  by  researchers. 

The  content  and  the  number  of  sections  in  the  instrument  may  be  modified 
depending  on  the  researcher’s  proposed  project  management  effectiveness  evaluation 
model.  For  example,  the  sections  of  risk  assessment  and  risk  control  may  be  combined 
together  under  the  title  risk  management  with  changes  in  the  content  of  the  sections. 
Then,  the  instrument  will  consist  of  fourteen  sections  instead  of  fifteen.  The  section  of 
project  planning  and  estimation  may  be  divided  into  two  sections  such  as  project 
planning  and  project  estimation.  Then,  the  instrument  will  consist  of  sixteen  sections 
instead  of  fifteen. 

The  software  project  management  evaluation  instrument,  SPMEI,  and  the 
evaluation  model  in  this  research  should  be  considered  separately.  Even  though  the 
instrument  and  the  evaluation  model  design  are  closely  related  to  each  other,  it  is  also 
possible  to  use  the  instrument,  SPMEI,  with  other  evaluation  models.  Such  uses  may  or 
may  not  necessitate  modifications  in  the  instrument  depending  on  the  research  goals  and 
research  design. 

7.  Pilot  Studies  on  the  Instrument 

Conducting  pilot  studies  has  indispensible  benefits.  Four  pilot  studies  were 
conducted  with  the  participation  of  one  project  manager,  one  technical  manager  and  two 
software  developers. 
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C.  SOFTWARE  PROJECT  MANAGEMENT  EVALUATION  MODEL 

(SPMEM) 

The  framework  for  software  project  management  guides  the  development  of  the 
evaluation  model.  The  project  management  areas  in  the  framework  have  a  one-to-one 
mapping  to  the  model  variables.  The  variables  in  the  high-level  model  are  people  area 
score  (PeopleS),  process  area  score  (ProcessS),  product  area  score  (Products),  and  risk 
area  score  (RiskS).  These  variables  correspond  to  the  main  areas  in  the  framework  that 
were  explained  in  the  Chapter  V  of  this  dissertation.  Furthermore,  the  coefficient  of  each 
variable  (in  other  words  the  associated  weight  of  each  variable)  is  identified  based  on  the 
results  of  the  survey. 

The  variables  in  the  high-level  model  are  calculated  based  on  the  responses  to  the 
SPMEI.  The  project  management  areas  (such  as  requirements  management,  risk 
assessment,  stakeholder  involvement,  etc.)  in  the  framework  are  categorized  under  one  of 
the  main  areas.  For  example,  teamwork  and  communication  is  categorized  under  the 
people  main  area.  Project  planning  and  estimation  area  is  categorized  under  the  process 
main  area,  and  so  forth.  For  each  of  these  project  management  areas,  there  is  an 
associated  model  to  determine  the  score  for  that  area.  These  scores  are  determined  based 
on  the  responses  to  software  project  management  evaluation  instrument  (SPMEI) 
questions. 

1.  High-Level  Evaluation  Model 

The  high-level  evaluation  model  for  the  metric  is  as  follows: 

PME  Score  =  PeopleS  4).  33  +ProcessS  x  0. 2907+ProciuctS  x  0.204+RiskSx  0.1753 

where: 

PME  Score:  Software  Project  Management  Effectiveness  Score, 

PeopleS:  People  Main  Area  Score, 

ProcessS:  Process  Main  Area  Score, 

Products:  Product  Main  Area  Score,  and 
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RiskS:  Risk  Main  Area  Score. 


The  people  main  area  score  (PeopleS)  is  calculated  as  follows: 

(C+T+L+OC+PM+Sl+S) 
People  Area  Score  = - — - 


where: 

C:  Communication  Area  Score, 

T:  Teamwork  Area  Score, 

L:  Leadership  Area  Score, 

OC:  Organizational  Commitment  Area  Score, 

PM:  Project  Management  Area  Score, 

SI:  Stakeholder  Involvement  Area  Score,  and 
S:  Staffing  and  Hiring  Area  Score. 

The  process  main  area  score  (ProcessS)  is  calculated  as  follows: 

( RM+  PMC+  PPE+SM) 
Process  Area  Score  = - 

4 


where: 

RM:  Requirements  Management  Area  Score, 

PMC:  Project  Monitoring  and  Control  Area  Score, 

PPE:  Project  Planning  and  Estimation  Area  Score,  and 
SM:  Scope  Management  Area  Score. 

The  product  main  area  score  (Products)  is  calculated  as  follows: 

( CM+QE ) 

Product  Area  Score  = - 

2 
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where: 


CM:  Configuration  Management  Score  and 
QE:  Quality  Engineering  Score. 

The  risk  main  area  score  (RiskS)  is  calculated  as  follows: 


Risk  Area  Score  = - 

2 

where: 

RA:  Risk  Assessment  Area  Score  and 

RC:  Risk  Control  Area  Score. 

2.  Project  Management  Area  Evaluation  Models 

The  software  project  management  evaluation  instrument  (SPMEI)  is  divided  into 

sections  based  on  the  project  management  areas  in  the  framework.  The  high-level  model 

uses  the  project  management  area  scores.  These  scores  are  determined  via  evaluation 

models  developed  for  each  area.  These  models  use  the  scores  derived  from  the 

participant’s  responses  to  SPMEI  questions  inquiring  about  the  project  management 

effectiveness.  Therefore,  for  each  response  or  responses  to  a  question  in  SPMEI,  there  is 

an  associated  score.  For  example,  the  responses  to  question  RM3  in  the  requirements 

management  section  is  scored  as  follows: 

RM3.  Which  of  the  following  activities  are  conducted  in  the 
project?  (Check  all  that  apply,) 

[  I  Market  surveys 

1X1  Customer/User  interviews 

1X1  Prototyping 

XI  Scenarios/  use  cases 

i  |  Observation  of  the  user  in  operation 

i  |  None 

In  the  example  above,  the  study  participant  indicated  that  they  have  conducted 
customer/user  surveys,  prototyping,  and  development  of  scenarios/use  cases  to  increase 
the  effectiveness  of  requirements  management  activities.  Each  response  to  this  question 
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has  a  score  of  “1”  except  for  the  response  of  “None”  which  is  associated  with  a  score  of 
“0.”  Since  the  study  participant  checked  three  of  the  responses,  the  total  score  for  this 
question  is  “3.” 


The  above  question,  RM3,  allows  for  multiple  responses.  Not  all  questions  in  the 
SPMEI  allow  for  multiple  responses.  Some  of  them  only  allow  for  one  response  among 


all  possible  responses.  For  example,  question  PPE4  in  the  project  planning  and  estimation 
section  allows  for  one  response  as  shown  below: 


PPE4.  Check  the  statement  that  applies  to  the  project.  (Check 
only  one.) 

I  I  The  project  plan  is  approved  by  the  stakeholders  such  as 
customers,  users,  project  team  members,  executive  management  etc. 
There  is  no  approval  process. 


In  the  example  above,  the  study  participant  indicated  that  there  is  no  approval 
process  for  the  project  plan.  This  response  is  associated  with  a  score  of  “-2,”  while  the 
other  response  is  associated  with  a  score  of  “2.”  In  this  example,  the  total  score  for  this 
question  to  be  used  in  the  evaluation  model  for  project  planning  and  estimation  section  is 


“-2. 
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Another  example  from  the  quality  engineering  section  is  as  follows: 


There  are  adequate  tools, 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

QE18 

equipment  and  resources  for 
testing. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

Applicable 

□ 

In  the  example  above,  the  study  participant  completely  disagreed  to  the  statement 
that  “there  are  adequate  tools,  equipment  and  resources  for  testing.”  The  score  associated 
with  this  response  is  “-2.” 


The  associated  scores  for  each  response  or  responses  to  the  questions  in  SPMEI 
are  provided  in  Appendix  G.  Adding  the  scores  associated  with  the  responses  in  each 
section  provides  an  initial  score  for  that  section.  Since  some  of  the  responses  are 
associated  with  a  negative  score,  it  is  possible  to  have  an  initial  score  for  a  section  lower 
than  zero.  In  order  to  shift  the  lowest  score  to  a  base  score  of  zero,  the  lowest  possible 
score  for  each  section  is  converted  to  positive  and  then  added  to  the  initial  score.  This 
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converted  lowest  score  is  called  the  shifting  factor.  Table  15  presents  the  number  of 
questions,  the  lowest  score,  the  highest  score  and  the  range  of  scores  for  each  project 
management  area.  Then,  this  shifted  score  is  normalized  to  a  scale  of  0  to  10  by 
multiplying  it  with  a  scaling  factor.  Table  16  provides  the  shifting  factors  and  scaling 
factors  for  each  project  management  area. 


PEOPLE  MAIN  AREA 

Number 

of 

Questions 

Lowest 

Score 

Highest 

Score 

Difference 
between  the 
highest  and 
lowest  score 

Communication 

23 

-38 

66 

104 

Teamwork 

30 

-54 

73 

127 

Leadership 

17 

-34 

34 

68 

Organizational  Commitment 

26 

-50 

62 

112 

Project  Manager 

27 

-52 

60 

112 

Stakeholder  Involvement  - 
Contract 

16 

-30 

42 

72 

Stakeholder  Involvement  - 
Market 

12 

-22 

34 

56 

Staffing  and  Hiring 

29 

-52 

64 

116 

PROCESS  MAIN  AREA 

Requirements  Management 

27 

-50 

73 

123 

Project  Monitoring  and  Control 

19 

-32 

54 

86 

Project  Planning  and  Estimation 

35 

-70 

104 

174 

Scope  Management 

16 

-26 

45 

71 

PRODUCT  MAIN  AREA 

Configuration  Management 

13 

-22 

38 

60 

Quality  Engineering 

20 

-36 

57 

93 

RISK  MAIN  AREA 

Risk  Assessment  -  No 
Subcontracting 

19 

-34 

57 

91 

Risk  Assessment  -  With 
Subcontracting 

20 

-38 

63 

101 

Risk  Control 

17 

-26 

28 

54 

Table  15.  Number  of  Questions,  Lowest  and  Highest  Possible  Score  and  Range  of  Scores 

for  each  Project  Management  Area 
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Project  Management  Area 

Shifting  Factor 

Scaling  Factor 

Communication 

38 

10 

104 

Teamwork 

54 

10 

127 

Leadership 

34 

10 

68 

Organizational  Commitment 

50 

10 

112 

Project  Manager 

52 

10 

112 

Stakeholder  Involvement  -  Contract 

30 

10 

72 

Stakeholder  Involvement  -  Market 

22 

10 

56 

Staffing  and  Hiring 

52 

10 

116 

Requirements  Management 

50 

10 

123 

Project  Monitoring  and  Control 

32 

10 

86 

Project  Planning  and  Estimation 

70 

10 

174 

Scope  Management 

26 

10 

71 

Configuration  Management 

22 

10 

60 

Quality  Engineering 

36 

10 

93 
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Risk  Assessment  -  No  Subcontracting 

34 

10 

91 

Risk  Assessment  -  With  Subcontracting 

38 

10 

101 

Risk  Control 

26 

10 

54 

Table  16.  Scaling  Factors  for  Project  Management  Areas 

The  steps  for  calculating  the  score  for  a  project  management  area  are  listed  as 
follows: 

1 .  Add  the  scores  for  each  response  in  a  section.  This  step  provides  an  initial 
score  for  a  project  management  area. 

2.  Add  the  shifting  factor  to  the  sum  of  the  scores  calculated  in  the  previous 
step.  This  step  provides  a  shifted  initial  score  for  a  project  management 
area. 

3.  Multiply  the  shifted  sum  of  scores  with  a  scaling  factor  to  normalize  the 
score  to  a  scale  of  0  to  10.  After  this  step,  the  project  management  area 
score  is  normalized  to  be  used  in  the  high-level  model. 

The  generic  model  to  determine  a  project  management  area  score  is: 


f 


Project  Management  Area  Score  =  Scaling  Factory,  y \PM \  +  Shifting  Factor 


V/=t 


In  the  model  above,  n  is  the  number  of  questions  for  a  specific  project  management  area. 
PMAI  is  the  score  computed  from  the  response  or  responses  to  a  specific  question.  For 
example,  in  the  communication  section  of  the  SPMEI,  there  are  twenty-three  questions. 

10 

Thus,  n  is  23  for  this  area  model.  For  the  communication  area,  the  scaling  factor  is  ^ 

and  the  shifting  factor  is  38.  The  identifiers  used  for  questions  in  each  section  are 
presented  in  Table  17. 
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Project  Management  Area  (PMA) 

Identifier 

Communication 

C 

Teamwork 

T 

Leadership 

L 

Organizational  Commitment 

OC 

Project  Manager 

PM 

Stakeholder  Involvement 

SI 

Staffing  and  Hiring 

S 

Requirements  Management 

RM 

Project  Monitoring  and  Control 

PMC 

Project  Planning  and  Estimation 

PPE 

Scope  Management 

SM 

Configuration  Management 

CM 

Quality  Engineering 

QE 

Risk  Assessment 

RA 

Risk  Control 

RC 

Table  17.  Identifiers  Corresponding  to  Project  Management  Areas 


a.  Communication  Area  Evaluation  Model 


Communication  Area  Score  =  x 

104 


f  n= 23 


A 


Zd+38 

V  i= 1 


6.  Teamwork  Area  Evaluation  Model 


Teamwork  Area  Score  = 


10_ 

127' 


f  n= 30 


Z?;+54 

V  /=! 


A 


J 
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c. 


Leadership  Area  Evaluation  Model 


In  the  leadership  section  of  SPMEI,  the  respondent  has  to  choose  to 
respond  to  one  of  two  questions:  L17  and  LI 8.  If  the  project  team  mostly  consists  of 
inexperienced  staff  then  the  respondent  should  answer  question  LI 7.  If  the  project  team 
mostly  consists  of  experienced  staff,  then  the  respondent  should  answer  question  LI 8. 
The  choices  for  these  questions  are  identical.  However,  the  scoring  is  different.  The 
model  for  both  cases  is  presented  below. 

If  the  team  mostly  consists  of  inexperienced  staff,  then  the  leadership  area 
model  is  as  follows: 


Leadership  Area  Score 


If  the  team  mostly  consists  of  experienced  staff,  then  the  leadership  area 
model  is  as  follows: 


Leadership  Area  Score 


C  n= 16  h 

Xa  +As +34 


V  i= 1  J 


d.  Organizational  Commitment  Evaluation  Area  Model 


Organizational  Commitment  Area  Score 


W_ 

112 


f  n= 26 


■X 


A 


+5o 

V  /= i  J 


e.  Project  Manager  Area  Evaluation  Model 


Project  Manage r  Area  Score 


W_ 

112 


f  n=21 


X 


A 


+52 

V  *=i  J 
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f.  Stakeholder  Involvement  Area  Evaluation  Model 


In  the  stakeholder  involvement  section  of  SPMEI,  the  questions  after  SI  10 
are  divided  into  two  sections.  If  the  project  is  developed  for  the  market  without  a  specific 
contract,  then  the  respondent  should  answer  questions  Sill  and  SI  12.  If  the  project  is 
developed  under  a  contract  with  a  customer,  then  the  respondent  should  not  answer  the 
questions  Sill  and  SI  12,  but  the  questions  from  SI  13  to  SI  18  instead. 

If  the  project  is  developed  for  the  market,  then  the  stakeholder 
involvement  area  model  is  as  follows: 


Stakeholdei' Involvement  Area  Score  =  —  x 

56 


f  n= 12 


2 >,  +22 

V  i= 1 


J 


If  the  project  is  developed  for  the  market,  then  the  stakeholder 
involvement  area  model  is  as  follows: 


10 


77=10 


18 


2 


Stakeholder  Involvement  Area  Score  =  — x|  I S,  +  Is, +30 

12 


7=1 


7=13 


g.  Staffing  and  Hiring  Area  Evaluation  Model 


Staffing  and  Hiring  Area  Score 


10 

~m 


f  77=29 


■X 


1$  +52 

V  i= 1 


2 


J 


h.  Requirements  Management  Area  Evaluation  Model 


Requirements  Management  Area  Score 


10_ 

123 


f  77=27 


X 


I*K  +50 

V  i= 1  J 
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Project  Monitoring  and  Control  Area  Evaluation  Model 


JQ  S'  n= 19  A 

Project  Monitoring  and  Control  Area  Score  =  — x|  ^ PMC.  +  32 


V  i~  1 


J 


Project  Planning  and  Estimation  Area  Evaluation  Model 


jq  r n=35  ^ 

Project  Planning  and  Estimation  Area  Score  =  -  —  xl£flfiE;+70 


k.  Scope  Management  Area  Evaluation  Model 


Scope  Management  Area  Score  =  ~fjx 


f  n= 16 


A 


V  /= i  J 


Configuration  Management  Area  Evaluation  Model 


10  ( ”=13  ^ 

Configuration  Management  Area  Score  =  —x  |  2_CMt+22 


V  *=i 


in.  Quality  Engineering  Area  Evaluation  Model 


Quality  Engineering  Area  Score 


n.  Risk  Assessment  Area  Evaluation  Model 

In  the  risk  assessment  section  of  the  SPMEI,  there  is  an  additional 
question  at  the  end  of  the  section  for  the  projects  in  which  subcontracting  is  used.  The 
question  identifier  is  RA20. 

If  the  project  does  not  utilize  subcontracting,  then  the  risk  assessment  area 
model  is  as  follows: 
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o.  Risk  Control  Area  Evaluation  Model 

In  the  risk  control  section  of  the  SPMEI,  there  are  four  questions  that  are 
excluded  from  the  evaluation  model:  RC1,  RC2,  RC3,  and  RC4.  These  questions  are 
included  in  the  instrument  to  enable  a  consistency  check  among  the  responses  and  for 
other  research  purposes.  Therefore,  for  the  risk  control  area  model,  only  the  responses 
from  RC5  to  RC17  are  included  in  the  evaluation  model: 


Risk  Control  Area  Score  = 


54 
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VIII.  SURVEY  STUDIES  ON  SOFTWARE  PROJECTS  AND  DATA 

ANALYSIS 


A.  INTRODUCTION 

In  order  to  test  the  hypothesis  stated  in  this  research,  data  from  real-world 
software  projects  were  collected.  The  data  were  collected  using  the  software  project 
management  evaluation  instrument  (SPMEI).  Additional  and  supplemental  data  is 
gathered  with  face-to-face,  phone  interviews,  or  electronic  correspondence  with  the  study 
participants.  The  study  participants  consisted  of  project  managers,  executive  managers, 
technical  managers,  project  team  leaders,  and  process/metrics  experts  who  worked  on  the 
projects.  An  essential  piece  of  data  collected  was  the  project  success  ratings  provided  by 
these  study  participants  for  their  software  development  projects.  The  responses  to  the 
SPMEI  and  the  evaluation  model  (SPMEM)  developed  were  used  to  measure  the  project 
management  effectiveness  in  software  projects.  The  dataset  consisted  of  two  metrics,  the 
project  success  rating  and  software  project  management  effectiveness  (PME)  metric. 
These  two  measures  were  used  to  test  the  research  hypothesis.  In  order  to  understand  the 
measure  of  association  between  these  two  metrics,  a  parametric  correlation  analysis  was 
conducted.  The  analysis  showed  that  there  is  a  strong  positive  correlation,  r,  between  the 
software  project  management  effectiveness  (PME)  metric  proposed  in  this  research  and 
the  software  project  success  rating  provided  by  the  study  participants. 

B.  DATA  COLLECTION 

The  project  data  collection  for  the  studies  was  one  of  the  most  challenging  parts 
of  this  research.  Even  though  the  author  gained  experience  in  surveying  methods  during 
validation  of  the  framework,  data  collection  for  the  validation  of  the  metric  presented  a 
number  of  new  and  different  challenges. 

Two  different  strategies  were  utilized  to  accomplish  data  collection.  The  first 
strategy  involved  calling  practitioners  for  participation  via  advertisements  in  magazines 
and  websites.  In  addition,  bulk  e-mails  are  sent  to  special  interest  groups.  Call  for 
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participation  announcements  were  posted  in  software  development  forum  (SDForum) 
newsletter,  software  technology  support  center  website  maintained  by  the  Air  Force, 
worldwide  software  process  improvement  network  initiated  by  Software  Engineering 
Institute,  software  development  forum  leadership  special  interest  group  network  and  in 
some  other  special  interest  group  networks.  Some  of  these  networks  reach  thousands  of 
software  practitioners.  However,  this  strategy  did  not  yield  effective  results.  Very  little 
participation  was  acquired  using  this  strategy.  It  is  observed  that  sharing  the  project  data 
with  a  researcher  requires  a  trust  relationship,  which  is  hard  to  establish  with  such  an 
advertisement-based  campaign. 

Since  the  first  strategy  was  found  to  be  ineffective,  a  new  strategy  for  data 
collection  was  devised.  The  new  strategy  involved  establishing  direct  communication  to 
software  managers  via  personal  relationships  or  via  acquaintances  of  friends  and 
colleagues.  This  strategy  yielded  much  better  results.  Most  of  the  participation  in  this 
research  was  acquired  using  this  strategy.  Direct  communication  with  possible  study 
participants  enables  easier  establishment  of  trust  relationships. 

The  author  conducted  interviews  with  study  participants  after  they  completed 
SPMEI.  The  goal  of  these  interviews  was  to  acquire  additional  insight  into  their  project 
development  efforts.  Not  all  the  participants  were  available  for  interviews  due  to  their 
busy  schedules.  However,  most  of  them  were  interviewed.  Whenever  possible,  these 
interviews  were  conducted  in  person.  When  a  face-to-face  interview  was  not  possible,  the 
interviews  were  conducted  either  by  phone  or  by  electronic  correspondence. 

C.  SURVEY  STUDIES 

In  this  section,  the  projects  analyzed  will  be  presented  one-by-one  in  detail.  The 
information  gathered  during  interviews  with  the  study  participants  were  also  presented 
briefly.  The  projects  were  given  identifiers  to  ensure  the  anonymity  of  the  responses. 
These  identifiers  are  letters  from  the  alphabet  assigned  to  each  project. 
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The  small  sample  size  limits  our  ability  to  perform  many  advanced  statistical 
analysis.  On  the  other  hand,  the  statistical  analyses  perfonned  provided  valuable 
information.  In  addition,  each  project  was  analyzed  in  detail.  These  in-depth  analyses 
revealed  additional  infonnation. 

As  indicated  by  many  scholars  and  practitioners,  project  management  is  complex 
by  nature.  Analysis  of  this  complex  endeavor  requires  careful  and  appropriate  selection 
of  research  methods.  In  this  research,  it  has  been  observed  that  in  order  to  develop  an 
effectiveness  measure  for  software  project  management,  a  broad  range  of  project 
management  areas  should  be  inquired  into.  Such  necessity  of  comprehensive  and  broad 
inquiry  led  to  conducting  in-depth  analysis  of  each  project  in  addition  to  conducting 
statistical  analysis  on  the  dataset.  The  software  project  management  evaluation 
instrument,  SPMEI,  gathers  data  from  a  software  project  for  over  500  data  points.  The 
amount  of  data  gathered  from  software  projects  is  much  more  than  the  amount  of  data 
gathered  from  projects  in  some  other  studies.  This  amount  of  data  enables  us  to  conduct 
detailed  analysis  for  each  project,  which  helped  to  test  the  hypothesis  stated  in  this  study. 

1.  Project  A 

The  goal  of  this  project  was  to  deliver  a  software  product  that  integrates  various 
functions  and  reports  of  various  information  management  systems  used  in  the  client’s 
company.  The  final  product  itself  was  another  information  management  system.  The 
project  is  considered  a  success  with  reservations  by  the  study  participant.  This  project 
was  conducted  by  a  small  software  company  in  the  U.S.A.  between  February  2006  and 
November  2006.  The  project’s  original  schedule  was  planned  as  seven  months,  while  the 
project  took  ten  months  to  complete.  All  of  the  functionality  agreed  upon  in  the  baseline 
was  delivered.  The  planned  budget  for  the  project  was  $500,000.  There  was  50%  cost 
overrun.  The  actual  cost  of  the  project  was  $750,000.  The  average  number  of  project 
team  members  involved  from  start  to  end  was  four.  The  technical  manager  of  this  project 
completed  the  SPMEI. 

This  study  was  initially  planned  as  a  pilot  study.  However,  after  the  study  was 
completed,  it  was  found  that  the  research  protocol  and  SPMEI  do  not  need  any  significant 
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modifications.  Only  a  few  of  the  questions  were  deleted  from  the  instrument,  while  very 
few  of  them  were  combined  together.  This  did  not  have  any  affect  on  the  evaluations. 
Therefore,  this  sample  is  included  in  the  main  study. 

After  the  participant  filled  out  the  instrument,  an  interview  was  conducted.  In  the 
interview,  the  technical  manager  was  asked  if  he  had  any  remarks  on  this  project.  The 
technical  manager  indicated  that  there  were  a  couple  of  problems  in  this  project.  These 
problems  led  the  late  delivery  of  the  product.  The  first  problem  was  that  executive 
management  dictated  the  schedule  and  budget;  they  were  merely  trying  to  win  the 
contract.  Even  though  the  technical  manager  opposed  the  schedule  from  the  start,  the 
project  was  a  go  with  its  original  schedule.  The  project  manager  felt  that  he  had  to  do 
what  he  could  with  what  he  had.  The  project  manager  did  not  oppose  the  schedule. 
During  the  project,  he  also  did  not  inform  executive  management  regarding  the  project 
problems.  Not  informing  executive  management  resulted  in  the  lack  of  involvement  from 
executive  management  when  needed.  The  second  biggest  problem  was  the 
underestimation  of  the  project  scope.  The  technical  manager  thought  the  discussions 
regarding  the  project  scope  were  cut  short.  The  project  was  undertaken  without  the 
necessary  initial  investigation.  On  the  other  hand,  the  project  was  well-planned  according 
to  its  estimated  scope.  The  third  problem  was  from  the  customer  side.  The  users  did  not 
get  involved  with  the  project  even  though  it  was  requested  by  the  project  team.  Since  one 
of  the  goals  of  the  project  was  to  automate  some  of  the  procedures  and  generate  reports 
for  the  customer’  organization,  the  project  team  needed  to  understand  these  procedures. 
The  project  team  requested  input  from  the  users,  for  example  to  help  identify  report 
contents,  formats,  etc.  The  project  team  could  not  get  the  necessary  input  in  time.  There 
were  some  hidden  factors  in  the  customer’s  organization  for  such  lack  of  necessary  user 
involvement.  There  were  some  political  and  procedural  issues  within  departments  in  the 
customer’s  organization  which  created  extra  challenges  for  the  project  team.  This  was 
another  significant  factor  for  late  delivery. 

The  technical  manager  rated  the  overall  success  of  the  project  with  a  “6.”  The 
SPMEI  score  of  the  project  was  “7.” 
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Timeframe  of  the  Proiect  A 

2006 

Location 

TJ.S.A. 

Actual  Cost/  Cost  Overrun 

S750K  /  50% 

Actual  Schedule/Schedule  Overrun 

10  Months  /  43% 

Functionalitv  Delivered 

100% 

Size  of  the  Product 

100  KLOC 

Average  Number  of  Peonle  involved  in  the  Proiect 

4 

Number  of  Peonle  Involved  in  Reauirements  Phase 

3 

Number  of  Peonle  Involved  in  Design  Phase 

2 

Number  of  Peonle  Involved  in  tmnlementation  Phase 

4 

Number  of  Peonle  Involved  in  Testing  and  Deliverv  Phase 

4 

Communication  Area  Score 

6.5 

Teamwork  Area  Score 

6.1 

Leadersh in  Area  Score 

5.9 

Organizational  Commitment  Area  Score 

5.4 

Proiect  Manager  Area  Score 

7.1 

Stakeholder  Involvement  Area  Score 

7.1 

Staffing  and  FTiring  Area  Score 

4.4 

Reauirements  Management  Area  Score 

7.0 

Proiect  Planning  and  Estimation  Area  Score 

6.4 

Proiect  Monitoring  and  Control  Area  Score 

7.6 

Scone  Management  Area  Score 

6.9 

Configuration  Management  Area  Score 

8.7 

Oualitv  Engineering  Area  Score 

7.1 

Risk  Assessment  Area  Score 

6.4 

Risk  Control  Area  Score 

6.3 

Peonle  Area  Score 

6.1 

Process  Area  Score 

7.0 

Product  Area  Score 

7.9 

Risk  Area  Score 

6.3 

PME  Score 

6.7 

Rounded  PME  Score 

7 

Proiect’s  Overall  Success  Rating 

6 

Table  18.  Project  A  Data  in  Tabular  Fonn 
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2.  Project  B 


The  goal  in  this  project  was  to  provide  a  DSL  CPE  firmware/software  upgrade  to 
a  commercially  available  DSL  CPE  chipset  and  reference  design.  The  study  participant 
was  the  executive  manager  who  oversaw  the  project. 

The  project  took  six  months  to  complete  and  took  place  in  2006.  The  project  was 
planned  to  be  completed  in  six  months,  thus  it  was  completed  on  time.  95%  of  the 
planned  functionality  was  delivered  with  the  project.  The  average  number  of  people 
involved  from  start  to  end  in  this  project  was  ten.  The  project  was  conducted  in  the 
U.S.A.  by  a  commercial  vendor.  The  study  participant  did  not  reveal  the  planned  and 
actual  effort,  nor  the  planned  budget  and  the  cost  of  the  project. 

After  the  study,  a  brief  interview  was  conducted  via  e-mail.  The  study  participant 
indicated  that  he  chose  this  project  because  it  was  completed  recently  and  he  could  still 
remember  most  of  the  details.  The  study  participant  is  generally  involved  with  Silicon- 
On-Chip  projects  that  include  a  portion  of  software  development.  He  mentioned  that  this 
was  a  typical  software  project  in  his  group. 

The  study  participant  rated  the  overall  success  of  the  project  with  a  “9.”  The  PME 
score  for  this  project  was  a  “7.” 
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Timeframe  of  the  Proiect  R 

2006 

Location 

TJ.S.A. 

Actual  Cost/  Cost  Overrun 

N/A 

Actual  Schedule/Schedule  Overrun 

6  Months  /  On  Time 

Functionalitv  Delivered 

95% 

Size  of  the  Product 

N/A 

Average  Number  of  Peonle  involved  in  the  Proiect 

10 

Number  of  Peonle  Involved  in  Reanirements  Phase 

4 

Number  of  Peonle  involved  in  Design  Phase 

10 

Number  of  Peonle  Involved  in  imnlementation  Phase 

15 

Number  of  Peonle  Involved  in  Testing  and  Deliverv  Phase 

10 

Communication  Area  Score 

7.2 

Teamwork  Area  Score 

7.8 

Leadersh in  Area  Score 

8.4 

Organizational  Commitment  Area  Score 

7.3 

Proiect  Manager  Area  Score 

7.9 

Stakeholder  Involvement  Area  Score 

6.6 

Staffing  and  FTiring  Area  Score 

7.3 

Reanirements  Management  Area  Score 

7.1 

Proiect  Planning  and  Estimation  Area  Score 

6.8 

Proiect  Monitoring  and  Control  Area  Score 

5.5 

Scone  Management  Area  Score 

7.0 

Configuration  Management  Area  Score 

7.2 

Oualitv  Engineering  Area  Score 

6.9 

Risk  Assessment  Area  Score 

5.6 

Risk  Control  Area  Score 

5.9 

Peonle  Area  Score 

7.5 

Process  Area  Score 

6.6 

Product  Area  Score 

7.0 

Risk  Area  Score 

5.8 

PME  Score 

6.8 

Rounded  PME  Score 

7 

Proiect’s  Overall  Success  Rating 

9 

Table  19.  Project  B  Data  in  Tabular  Form 
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3.  Project  C 


This  study  was  different  in  one  important  aspect  from  other  projects  in  this  data 
set.  When  this  survey  was  conducted,  the  project  was  still  in  progress.  Study  results 
support  the  case  for  the  use  of  PME  and  the  evaluation  model  as  a  monitoring  tool.  The 
model  successfully  evaluated  the  project  management  effectiveness  in  this  project  as  it 
was  intended. 

The  goal  of  this  project  was  to  develop  a  small  size  medical  instrument.  A  portion 
of  the  project  involves  embedded  software  development  and  a  PC  installer  application  for 
this  medical  instrument.  The  study  was  conducted  not  just  on  the  software  development 
part  but  also  on  the  whole  project.  This  project  started  in  September  2007  and  is  expected 
to  finalize  in  February  2009  with  schedule  slippages.  Originally,  it  was  planned  to  be 
completed  in  twelve  months.  However,  the  project  schedule  is  already  slipping  and  at  the 
time  of  study,  it  was  expected  to  finish  in  seventeen  months.  The  study  took  place  during 
the  seventh  month  of  the  development.  The  overall  projected  effort  for  the  project  was 
125  man-months.  At  the  time  of  evaluation,  the  expected  actual  effort  was  175  man- 
months.  For  the  software  development  effort,  the  projected  effort  was  ten  man-months 
and  the  expected  actual  effort  was  twenty  man-months.  The  planned  budget  for  the 
project  was  $3.5  million.  At  the  time  of  evaluation,  the  expected  actual  cost  was  $4.5 
million  with  the  changes  occurring  in  the  project.  For  the  software  development  effort, 
the  projected  budget  was  $200,000  and  the  expected  actual  cost  became  $400,000.  At  the 
time  of  study,  one-third  of  the  expected  functionality  was  completed.  The  project  team 
expects  to  deliver  100%  functionality.  In  total,  eighteen  people  are  involved  in  this 
project,  with  the  average  being  ten.  For  the  software  development  effort,  four  people  are 
involved  in  the  project  with  the  average  being  four  people.  This  project  is  being 
conducted  in  the  U.S.A.  with  most  of  the  team  being  geographically  dispersed.  Most  of 
the  project  team  is  composed  of  contractors  working  temporarily  on  this  project.  This 
project  is  conducted  in  a  commercial  environment  and  the  company  funding  the  project 
expects  to  distribute  the  developed  medical  instrument  to  the  market  in  different 
countries. 
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In  this  study,  the  participant  was  an  executive  manager  who  was  overseeing  the 
project.  He  was  an  experienced  manager  with  more  than  twenty  years  of  experience  in 
software  development  projects.  The  author  was  with  the  study  participant  while  he  was 
completing  the  SPMEI  for  the  project  he  was  overseeing  at  the  time.  The  author  merely 
observed  the  study  participant  and  did  not  interfere  with  the  procedure.  It  was  observed 
that  the  study  participant  did  not  have  any  difficulty  in  responding  to  the  statements  and 
questions  in  SPMEI.  After  the  participant  completed  SPMEI,  a  comprehensive  interview 
was  conducted  on  the  project  and  project  management  challenges. 

A  major  challenge  in  this  project  was  overcoming  the  bureaucratic  procedures  to 
get  the  medical  instrument  approved  by  regulatory  agencies.  Preparation  and  submission 
of  the  necessary  documents  and  waiting  for  responses  were  all  challenging  tasks  in  the 
project  even  though  they  hired  experts  and  consultants  for  these  issues.  The  responsible 
regulatory  agency  in  the  U.S.  is  the  Federal  Drug  Administration  (FDA).  Since  the 
company  is  also  planning  to  distribute  the  medical  instrument  in  other  countries,  they  also 
needed  to  comply  with  medical  regulations  in  other  countries.  Naturally,  regulations  are 
not  the  same  for  all  the  countries.  The  study  participant  attributed  the  reason  for  the 
schedule  slip  partially  to  these  issues.  In  order  to  finalize  some  of  the  specifications,  they 
needed  to  wait  for  certain  approvals  from  these  regulatory  agencies.  Because  of  these 
issues,  the  requirements  in  this  project  have  not  reached  stability  at  the  time  of  the  study. 

During  the  interview,  the  study  participant  mentioned  that  they  recently 
discovered  an  important  issue.  In  their  last  project  meeting,  they  realized  that  the  project 
was  suffering  from  not  having  a  shared  vision;  not  all  project  team  members  were  sharing 
the  same  vision  for  the  project.  They  were  having  discussions  on  how  to  make  the  project 
vision  more  clear  and  ensuring  that  it  is  understood  by  the  project  team  members. 

Another  issue  they  had  just  realized  during  the  project  was  that  notable  project 
accomplishments/milestones/deliverables  were  not  celebrated  with  social  events  and 
parties.  This  became  an  issue  because  the  developers  would  like  to  see  such  social  events 
in  this  project.  This  issue  is  specifically  addressed  in  PME  with  the  question  T3.  At  the 
time  of  study,  the  executive  management  was  reviewing  how  to  distribute  the  bonuses  to 
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project  team  members.  This  issue  was  particularly  important  for  them  since  most  of  the 
team  members  are  contractors  and  they  believed  this  would  be  a  significant  factor  on  the 
project  performance. 

The  study  participant  made  a  controversial  remark  during  the  study.  He  said  that, 
“the  project  plan  changes  regularly  so  I  am  not  sure  whether  it  is  better  than  no  plan.”  All 
project  plans  change.  This  is  almost  a  fact  for  project  management.  This  case  was  no 
different.  The  controversy  in  his  remark  was  that  the  project  plan  was  changing  so  much 
that  it  was  questionable  whether  there  was  any  use  for  it.  He  also  indicated  that  even 
though  there  was  a  formal  documented  plan,  the  updates  had  been  informal. 

The  study  participant  rated  the  current  success  of  the  project  with  a  “5”  at  the  time 
of  the  study.  The  PME  score  for  this  project  was  a  “6.”  The  study  participant  assessed  the 
status  of  the  project  as  a  “painful  success.”  Even  though  the  project  was  having 
difficulties  in  project  management,  the  participant  was  expecting  a  significant  project 
success  from  the  business  perspective. 
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Timeframe  of  the  Proiect  C 

2007  -  2009 

Location 

TJ.S.A. 

Actual  Cost/  Cost  Overrun 

$4.5  Million  /  28.5% 

Actual  Schedule/Schedule  Overrun 

17  Months  /  42% 

Functionalitv  Delivered 

100%  (exnectedl 

Size  of  the  Product 

100  KLOC 

Average  Number  of  Peonle  involved  in  the  Proiect 

1 0  (  Whole  Proiect! 

Number  of  Peonle  Involved  in  Reanirements  Phase 

3  (SW  nortionl 

Number  of  Peonle  involved  in  Design  Phase 

3  (SW  nortionl 

Number  of  Peonle  Involved  in  imnlementation  Phase 

4  (SW  nortionl 

Number  of  Peonle  Involved  in  Testing  and  Deliverv  Phase 

4  (SW  nortionl 

Communication  Area  Score 

6.4 

Teamwork  Area  Score 

6.1 

Leadersh in  Area  Score 

6.2 

Organizational  Commitment  Area  Score 

7.6 

Proiect  Manager  Area  Score 

5.2 

Stakeholder  Involvement  Area  Score 

5.7 

Staffing  and  FTiring  Area  Score 

5.9 

Reanirements  Management  Area  Score 

7.2 

Proiect  Planning  and  Estimation  Area  Score 

5.6 

Proiect  Monitoring  and  Control  Area  Score 

6.2 

Scone  Management  Area  Score 

6.1 

Configuration  Management  Area  Score 

4.5 

Oualitv  Engineering  Area  Score 

7.8 

Risk  Assessment  Area  Score 

5.5 

Risk  Control  Area  Score 

4.4 

Peonle  Area  Score 

6.4 

Process  Area  Score 

6.3 

Product  Area  Score 

6.2 

Risk  Area  Score 

5.0 

PME  Score 

6.1 

Rounded  PME  Score 

6 

Proiect’s  Overall  Success  Rating 

5 

Table  20.  Project  C  Data  in  Tabular  Form 
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4.  Project  D 


The  goal  in  this  project  was  to  create  a  prototype  of  software  which  was  to  be  run 
on  a  stand-alone  device.  The  prototype  would  be  a  technical  proof-of-concept  to  be  used 
for  demos  and  presentations  to  senior  management  and  other  stakeholders.  A  Java 
application  was  developed  using  an  XML  file  for  configuration  and  Derby  as  a  database. 
Deliverables  were  the  software  application  as  an  installer,  installation  and  setup  guide, 
and  user  manual.  The  study  participant  was  a  project  team  leader  in  this  project. 

This  project  was  conducted  between  July  2006  and  November  2006.  The  project 
was  planned  for  completion  in  six  months  but  was  completed  in  just  four  months,  so  the 
project  was  completed  earlier  than  expected.  The  planned  budget  for  the  project  was 
$29,000  and  the  actual  cost  was  $20,000.  The  projected  effort  was  twenty-four  man- 
months,  while  the  actual  effort  was  sixteen  man-months.  The  project  team  delivered 
100%  functionality.  The  average  number  of  people  involved  in  the  project  was  five.  The 
lines  of  code  in  the  final  product  totaled  30,000. 

This  project  was  different  from  other  projects  in  this  data  set  in  one  aspect:  it  was 
developed  by  project  team  members  who  were  geographically  dispersed  across  several 
countries.  The  project  was  developed  by  team  members  from  the  U.S.,  UK  and  India. 
This  project  was  developed  in  a  commercial  environment. 

An  interview  could  not  be  conducted  with  the  participant  after  the  survey  study. 
The  participant  also  did  not  provide  any  special  remarks  on  this  project.  The  participant 
was  quite  concerned  with  the  anonymity  of  the  project  data. 

The  study  participant  rated  the  overall  success  of  the  project  with  a  “7.”  The  PME 
score  for  this  project  was  a  “6.” 
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Timeframe  of  the  Proiect  D 

2006 

Location 

U  S  A..  UK.  India 

Actual  Cost/  Cost  Overrun 

$20K  /  Under  Budget 

Actual  Schedule/Schedule  Overrun 

4  Months  /  On  Time 

Functionalitv  Delivered 

100% 

Size  of  the  Product 

30  KEOC 

Average  Number  of  Peonle  involved  in  the  Proiect 

5 

Number  of  Peonle  Involved  in  Reauirements  Phase 

4 

Number  of  Peonle  Involved  in  Design  Phase 

4 

Number  of  Peonle  Involved  in  tmnlementation  Phase 

5 

Number  of  Peonle  Involved  in  Testing  and  Deliverv  Phase 

5 

Communication  Area  Score 

7.1 

Teamwork  Area  Score 

7.1 

Leadersh in  Area  Score 

6.6 

Organizational  Commitment  Area  Score 

6.7 

Proiect  Manager  Area  Score 

8.1 

Stakeholder  Involvement  Area  Score 

7.7 

Staffing  and  FTiring  Area  Score 

5.9 

Reauirements  Management  Area  Score 

5.4 

Proiect  Planning  and  Estimation  Area  Score 

6.7 

Proiect  Monitoring  and  Control  Area  Score 

6.9 

Scone  Management  Area  Score 

5.9 

Configuration  Management  Area  Score 

2.2 

Oualitv  Engineering  Area  Score 

5.6 

Risk  Assessment  Area  Score 

5.5 

Risk  Control  Area  Score 

5.7 

Peonle  Area  Score 

7.0 

Process  Area  Score 

6.2 

Product  Area  Score 

3.9 

Risk  Area  Score 

5.6 

PME  Score 

5.9 

Rounded  PME  Score 

6 

Proiect’s  Overall  Success  Rating 

7 

Table  2 1 .  Project  D  Data  in  Tabular  Fonn 
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5.  Project  E 


The  goal  in  this  project  was  to  create  and  deliver  a  system  that  would  perform 
sales  order  management  and  “pick  and  pack”  operations  in  the  packaging  and  distribution 
center,  while  creating  a  new  hardware  and  network  infrastructure  for  the  client’s  IT 
people.  The  study  participant  was  the  quality  and  metrics  subject  matter  expert  of  the 
project. 

This  project  suffered  a  significant  schedule  slip.  The  schedule  overrun  was  267%. 
The  project  was  originally  planned  for  completion  in  nine  months.  However,  it  took 
twenty-four  months  to  complete.  The  project  was  developed  between  August  1993  and 
July  1995.  The  projected  effort  was  180  man-months,  while  the  actual  became  360  man- 
months.  In  addition,  only  70%  of  the  functionality  in  the  initial  baseline  was  delivered. 
The  product  size  was  over  than  10,000  function  points.  The  total  number  of  people 
involved  in  this  development  effort  was  forty-three.  The  participant  did  not  reveal  the 
budget  and  the  cost  of  the  project.  This  project  was  developed  by  a  major  commercial 
software  development  company  in  the  U.S. 

A  brief  interview  was  conducted  with  the  study  participant  after  the  completion  of 
PME.  The  study  participant  viewed  the  project  as  a  failure  even  though  the  company 
delivered  the  project.  The  participant  indicated  that  the  root  cause  for  the  failure  of  the 
project  was  that  the  chief  architect  ran  away  with  the  project.  The  project  became  his,  not 
the  project  manager’s  project. 

The  study  participant  rated  the  overall  success  of  the  project  with  a  “3.”  The  PME 
score  for  this  project  was  a  “5.” 
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Timeframe  of  the  Proiect  E 

1 993-1995 

Location 

TJ.S.A. 

Actual  Cost/  Cost  Overrun 

N/A 

Actual  Schedule/Schedule  Overrun 

24  Months  /  167% 

Functionalitv  Delivered 

70% 

Size  of  the  Product 

1 0.000 +  FP 

Average  Number  of  Peonle  involved  in  the  Proiect 

N/A 

Number  of  Peonle  Involved  in  Reauirements  Phase 

4 

Number  of  Peonle  involved  in  Design  Phase 

4 

Number  of  Peonle  Involved  in  imnlementation  Phase 

20 

Number  of  Peonle  Involved  in  Testing  and  Deliverv  Phase 

15 

Communication  Area  Score 

6.3 

Teamwork  Area  Score 

7.1 

Eeadershin  Area  Score 

5.0 

Organizational  Commitment  Area  Score 

8.1 

Proiect  Manager  Area  Score 

6.7 

Stakeholder  Involvement  Area  Score 

5.7 

Staffing  and  ETiring  Area  Score 

6.1 

Reauirements  Management  Area  Score 

3.8 

Proiect  Planning  and  Estimation  Area  Score 

7.1 

Proiect  Monitoring  and  Control  Area  Score 

5.2 

Scone  Management  Area  Score 

4.2 

Configuration  Management  Area  Score 

2.2 

Oualitv  Engineering  Area  Score 

7.1 

Risk  Assessment  Area  Score 

3.7 

Risk  Control  Area  Score 

3.7 

Peonle  Area  Score 

6.4 

Process  Area  Score 

5.1 

Product  Area  Score 

4.6 

Risk  Area  Score 

3.7 

PME  Score 

5.2 

Rounded  PME  Score 

5 

Proiect’s  Overall  Success  Rating 

3 

Table  22.  Project  E  Data  in  Tabular  Form 
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6.  Project  F 


The  goal  in  this  project  was  to  create  a  new  version  of  an  existing  contact  center 
application  for  sales  to  customers  throughout  the  world.  The  application  is  based  on  C++ 
and  Java,  and  has  elements  that  involve  VoIP  telephony,  as  well  as  thin-  and  thick- 
clients.  The  deliverables  of  the  project  included  the  application,  documentation,  and 
statistical  information  to  the  business  partner,  including  defect  trends,  test  reports,  etc. 
The  study  participant  was  the  program  and  engineering  manager  of  the  project. 

This  project  took  place  between  July  2007  and  July  2008.  The  original  schedule 
planned  completion  in  ten  months.  However,  the  project  took  twelve  months  to  complete. 
The  projected  effort  in  this  project  was  220  man-months  while  the  actual  effort  was  275 
man-months.  The  study  participant  did  not  reveal  budget  and  cost  data  for  the  project 
because  of  privacy  reasons.  The  company  delivered  95%  of  the  functionality  in  the  initial 
baseline.  The  average  number  of  people  involved  in  this  project  was  twenty.  This  project 
was  developed  by  a  commercial  company  in  the  U.S. 

An  interview  could  not  be  conducted  with  the  participant  after  the  study.  Also,  the 
participant  did  not  provide  any  special  remarks  on  this  project. 

The  study  participant  rated  the  overall  success  of  the  project  with  a  “7.”  The  PME 
score  for  this  project  was  a  “6.” 
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Timeframe  of  the  Proiect  F 

2007-2008 

Location 

TJ.S.A. 

Actual  Cost/  Cost  Overrun 

N/A 

Actual  Schedule/Schedule  Overrun 

12  Months  /  20% 

Functionalitv  Delivered 

95% 

Size  of  the  Product 

N/A 

Average  Number  of  Peonle  Involved  in  the  Proiect 

20 

Number  of  Peonle  Involved  in  Reauirements  Phase 

5 

Number  of  Peonle  Involved  in  Design  Phase 

14 

Number  of  Peonle  Involved  in  tmnlementation  Phase 

16 

Number  of  Peonle  Involved  in  Testing  and  Deliverv  Phase 

22 

Communication  Area  Score 

6.1 

Teamwork  Area  Score 

6.4 

Leadersh in  Area  Score 

7.9 

Organizational  Commitment  Area  Score 

6.4 

Proiect  Manager  Area  Score 

7.7 

Stakeholder  Involvement  Area  Score 

3.4 

Staffing  and  FTiring  Area  Score 

6.3 

Reauirements  Management  Area  Score 

4.8 

Proiect  Planning  and  Estimation  Area  Score 

4.9 

Proiect  Monitoring  and  Control  Area  Score 

6.6 

Scone  Management  Area  Score 

4.9 

Configuration  Management  Area  Score 

4.0 

Oualitv  Engineering  Area  Score 

7.2 

Risk  Assessment  Area  Score 

5.6 

Risk  Control  Area  Score 

5.4 

Peonle  Area  Score 

6.3 

Process  Area  Score 

5.3 

Product  Area  Score 

5.6 

Risk  Area  Score 

5.5 

PME  Score 

5.7 

Rounded  PME  Score 

6 

Proiect’s  Overall  Success  Rating 

7 

Table  23.  Project  F  Data  in  Tabular  Form 
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7. 


Project  G 


The  goal  in  this  project  was  to  deliver  an  upgrade  for  a  command  and  control 
system  for  an  air  training  range.  Real-time  software  was  developed  and  delivered.  The 
study  participant  was  the  technical  manager  of  this  project. 

The  project  took  place  between  September  1982  and  July  1983.  The  projected 
schedule  was  ten  months  and  the  project  was  completed  in  ten  months.  In  this  project, 
there  was  going  to  be  a  schedule  slip;  however,  the  functionality  was  cut  down  and  the 
project  was  delivered  at  the  end  of  the  schedule.  70%  of  the  functionality  from  the  initial 
baseline  was  delivered  in  this  project.  The  projected  and  the  actual  cost  data  was  not 
revealed  by  the  study  participant.  The  projected  effort  in  this  project  was  96  man-months 
and  the  actual  effort  was  96  man-months.  The  average  number  of  people  involved  in  this 
project  was  ten.  16,000  lines  of  code  were  delivered  with  the  product.  This  project  was 
developed  by  a  major  software  company  in  the  U.S. 

A  brief  interview  was  conducted  with  the  study  participant  after  the  completion  of 
SPMEI.  The  participant  indicated  that  this  was  an  interesting  project  from  a  project 
management  perspective.  At  the  start  of  the  project,  even  though  the  entire  technical  team 
objected  to  the  budget  and  schedule,  the  company  management  signed  up  to  deliver  all  of 
the  functionality  and  other  deliverables  the  customer  wanted.  The  technical  team  assured 
the  management  it  was  not  possible  to  deliver  all  functionality  requested  within  the 
budget  and  schedule  given.  As  a  technical  manager,  the  participant  was  criticized  for 
being  behind  in  project  status  meetings  the  first  several  months.  There  were  three  tasks 
out  of  ten  the  technical  team  had  not  started  to  work  on.  By  the  fourth  to  fifth  months  into 
the  contract,  the  customer  had  failed  to  deliver  the  needed  interface  control  documents  for 
interfaces  the  three  tasks  required.  These  tasks  were  the  ones  the  project  team  had  not 
started  to  work  on.  By  the  end  of  the  contract,  the  customer  was  happy  with  the 
functionality  delivered,  and  agreed  that  their  failure  to  deliver  the  needed  interface 
control  documents  was  an  adequate  basis  for  the  non-completion  of  three  of  the  ten 
tasks.  The  other  seven  tasks  were  delivered  successfully. 
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The  study  participant  rated  the  overall  success  of  the  project  with  an  “8.”  The 
PME  score  for  this  project  was  “7.” 


Timeframe  of  the  Project  G 

1982-1983 

Location 

U.S.A. 

Actual  Cost/  Cost  Overrun 

N/A 

Actual  Schedule/Schedule  Overrun 

10  Months  /  On  Time 

Functionality  Delivered 

70% 

Size  of  the  Product 

16KLOC 

Average  Number  of  People  Involved  in  the  Project 

10 

Number  of  People  Involved  in  Requirements  Phase 

8 

Number  of  People  Involved  in  Design  Phase 

10 

Number  of  People  Involved  in  Implementation  Phase 

10 

Number  of  People  Involved  in  Testing  and  Delivery  Phase 

10 

Communication  Area  Score 

5.0 

Teamwork  Area  Score 

5.9 

Leadership  Area  Score 

5.9 

Organizational  Commitment  Area  Score 

5.7 

Project  Manager  Area  Score 

6.6 

Stakeholder  Involvement  Area  Score 

5.4 

Staffing  and  Hiring  Area  Score 

6.6 

Requirements  Management  Area  Score 

7.0 

Project  Planning  and  Estimation  Area  Score 

6.3 

Project  Monitoring  and  Control  Area  Score 

7.0 

Scope  Management  Area  Score 

6.1 

Configuration  Management  Area  Score 

8.2 

Quality  Engineering  Area  Score 

7.5 

Risk  Assessment  Area  Score 

6.8 

Risk  Control  Area  Score 

5.9 

People  Area  Score 

5.9 

Process  Area  Score 

6.6 

Product  Area  Score 

7.8 

Risk  Area  Score 

6.4 
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PME  Score 

6.6 

Rounded  PME  Score 

7 

Project’s  Overall  Success  Rating 

8 

Table  24.  Project  G  Data  in  Tabular  Fonn 


8.  Project  H 

The  goal  in  this  project  was  to  develop  Navy  command  control  software  for 
service  oriented  based  object  management  and  data  fusion  over  low  bandwidth  IP-based 
data  links.  The  study  participant  was  the  project  manager  of  the  project. 

The  project  was  developed  between  July  2003  and  July  2005.  The  projected 
schedule  was  twenty-four  months  and  the  project  took  twenty-four  months  to  complete, 
so  the  project  did  not  suffer  from  a  schedule  overrun.  The  projected  budget  was  $3.2 
million  and  the  cost  of  the  project  was  the  same.  There  was  a  slight  overrun  in  terms  of 
effort:  the  projected  effort  was  200  man-months,  while  the  actual  effort  was  230  man- 
months.  90%  of  the  functionality  from  the  initial  baseline  was  delivered  at  the  end.  The 
average  number  of  people  involved  in  this  project  was  eight.  The  project  size  was  10,000 
lines  of  code  and  fifty  function  points.  This  project  was  developed  by  a  government 
organization  in  the  U.S. 

A  brief  interview  was  conducted  with  the  participant  after  this  survey  study.  The 
participant  viewed  the  project  as  a  success  overall.  The  participant  indicated  that  they 
used  very  expensive  and  talented  developers  in  this  project.  They  also  kept  the  team  size 
small  and  kept  the  requirements  tight.  In  this  project,  they  were  challenged  by  keeping 
the  developers  interacting  with  the  users.  A  unique  aspect  of  this  project  is  that  the 
system  ended  up  transitioning  for  a  different  user  than  originally  intended. 

The  study  participant  rated  the  overall  success  of  the  project  with  a  “7.”  The  PME 
score  for  this  project  was  “6.” 
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Timeframe  of  the  Proiect  FI 

2003-2005 

Location 

TJ.S.A. 

Actual  Cost/  Cost  Overrun 

S3. 2  Million  /  None 

Actual  Schedule/Schedule  Overrun 

24  Months  /  On  Time 

Functionalitv  Delivered 

90% 

Size  of  the  Product 

10KLOC/50FP 

Average  Number  of  Peonle  involved  in  the  Proiect 

8 

Number  of  Peonle  Involved  in  Reauirements  Phase 

2 

Number  of  Peonle  Involved  in  Design  Phase 

2 

Number  of  Peonle  Involved  in  tmnlementation  Phase 

4 

Number  of  Peonle  Involved  in  Testing  and  Deliverv  Phase 

5 

Communication  Area  Score 

6.2 

Teamwork  Area  Score 

5.7 

Leadersh in  Area  Score 

5.9 

Organizational  Commitment  Area  Score 

5.3 

Proiect  Manager  Area  Score 

6.4 

Stakeholder  Involvement  Area  Score 

7.2 

Staffing  and  FTiring  Area  Score 

4.6 

Reauirements  Management  Area  Score 

5.3 

Proiect  Planning  and  Estimation  Area  Score 

6.2 

Proiect  Monitoring  and  Control  Area  Score 

5.1 

Scone  Management  Area  Score 

.3.7 

Configuration  Management  Area  Score 

5.5 

Oualitv  Engineering  Area  Score 

5.5 

Risk  Assessment  Area  Score 

5.0 

Risk  Control  Area  Score 

6.3 

Peonle  Area  Score 

5.9 

Process  Area  Score 

5.1 

Product  Area  Score 

5.5 

Risk  Area  Score 

5.7 

PME  Score 

5.5 

Rounded  PME  Score 

6 

Proiect’s  Overall  Success  Rating 

7 

Table  25.  Project  H  Data  in  Tabular  Fonn 
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9.  Project  I 


The  goal  in  this  project  was  to  automate  100  million  paper  records  in  U.S. 
telephone  companies  and  streamline  administrative  functions.  This  was  a  very  large-scale 
project.  The  study  participant  was  the  project  manager  of  the  project. 

This  project  was  the  biggest  project  in  this  data  set  in  terms  of  cost,  scope  and 
number  of  people  involved  in  its  development.  It  was  considered  a  big  success  by  the 
project  manager.  The  developed  system  was  successfully  used  for  twenty  years.  The  cost 
of  the  project  was  $1  billion  and  it  saved  $1  billion  every  year.  The  return  on  investment 
from  this  project  was  multiplied  every  year.  In  addition,  it  has  the  highest  PME  score 
within  this  data  set. 

This  project  was  undertaken  between  1980  and  1983.  It  was  one  of  the  biggest 
projects  of  its  time.  When  the  project  started,  it  was  not  even  known  whether  it  could  be 
accomplished  or  not.  There  had  been  five  attempts  on  the  project  before  and  all  of  them 
failed.  The  projected  schedule  was  thirty-six  months.  At  the  end  of  the  schedule,  it  was 
understood  that  additional  functionality  was  needed  for  the  system  to  be  used  effectively. 
Therefore,  the  schedule  was  extended  to  forty-eight  months.  At  the  end  of  this  updated 
schedule,  because  of  the  functionality  added  to  the  baseline,  the  overall  functionality 
delivered  became  150%  when  compared  to  initial  baseline.  The  projected  effort  for  this 
project  was  1,000  man-months.  The  total  cost  of  the  project  was  $1  billion.  The  average 
number  of  people  involved  in  this  development  effort  was  300.  This  project  was 
developed  in  the  U.S.  and  it  became  the  infrastructure  for  the  telephone  companies  in  the 
U.S. 

A  brief  interview  was  conducted  with  the  project  manager.  In  addition,  the  study 
participant  provided  the  author  with  a  report  on  the  project.  This  report,  dated  2000,  was 
a  detailed  analysis  of  the  project  prepared  in  a  semester  by  four  students.  Therefore,  there 
was  abundant  information.  The  study  participant  indicated  that  this  project  was  a  big 
success  and  it  saved  $1  billion  a  year  after  it  was  in  use.  The  project  manager  and  his 
second  level  managers  were  well  rewarded  with  promotions  and  other  compensation.  In 
this  project,  risks  were  managed  with  a  semi-formal  action  item  tracking  and  each  item 
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was  carefully  tracked.  Configuration  management  was  executed  by  a  special  group  of 
software  manufacturers.  The  findings  of  the  report  provided  more  detail  on  the  project. 
Because  of  the  earlier  failed  attempts  on  the  project,  the  company  knew  the  complexity 
and  the  uncertainty  surrounding  this  project.  They  took  the  project  very  seriously  from 
the  start.  This  project  was  so  important  that  the  survival  of  the  customer  depended  on  its 
successful  completion  and  deployment.  Therefore,  cost  was  not  an  issue  and  schedule 
was  only  an  issue  to  a  certain  extent.  Full  delivery  of  the  necessary  functionality  and 
successful  operation  of  the  deployed  systems  were  of  the  utmost  importance  for  survival. 
The  project  manager  indicated  that  because  of  this  he  was  allowed  to  have  as  many 
people  and  resources  as  he  needed  at  his  disposal  in  this  project.  The  customer  was  a 
telephone  company  in  this  project.  The  customer’s  operations  were  manual  at  the  time 
and  these  manual  operations  were  no  longer  able  to  sustain  the  needs  of  the  company. 

Effective  communication  was  an  important  success  factor  in  this  project.  There 
were  bi-weekly  meetings  with  broad  inclusion  of  managers,  executive  management, 
customer,  users,  and  project  team  members.  In  the  weeks  that  there  were  no  meetings,  the 
project  manager  and  his  development  managers  published  a  project  newsletter.  This  bi¬ 
weekly  newsletter  infonned  all  interested  parties  of  the  hot  topics  and  important  issues  of 
the  project.  The  project  manager,  including  development  managers,  nourished  an  open 
project  environment  in  which  project  problems  were  openly  reported  and  discussed. 

At  the  beginning  of  the  project,  there  were  two  separate  teams  for  the 
development  of  the  project.  One  of  the  teams  had  competent  and  skillful  management  but 
lacked  talented  developers  with  the  necessary  skills  for  the  project.  The  other  team  was 
just  the  opposite,  and  was  composed  of  talented  and  skilful  developers  with  poor 
management.  The  project  manager  established  a  project  organization  out  of  these  two 
teams  by  switching  the  necessary  people  to  form  a  team  of  talented,  skillful,  and 
competent  managers  and  technicians. 

Effective  communication  also  brought  the  support  of  executive  management  and 
users.  The  deployed  system  was  adapted  quickly  by  the  users  since  their  inputs  were 
valued  from  the  start  of  the  project.  The  need  for  survival  also  brought  support. 
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This  project  was  quite  a  challenging  one  which  had  been  attempted  earlier  and 
failed.  The  project  team  was  aware  of  this  challenge.  There  were  also  unproven 
technologies  involved  in  the  development.  This  was  the  first  time  relational  databases 
were  to  be  used  in  such  a  commercial  project.  This  made  the  project  interesting  for  the 
talented  developers.  There  were  social  considerations  as  well.  Involvement  in  a  project  of 
this  scale  and  the  successful  completion  of  it  may  be  considered  a  reward. 

The  project  managers  and  other  managers  in  this  project  were  quite  talented  and 
competent.  They  protected  the  team  from  outside  interference  and  unnecessary  politics. 
They  employed  various  techniques  to  motivate  the  team.  One  of  them  was  “punishment 
through  embarrassment.”  Whenever  a  project  team  member  could  not  accomplish  what 
he  was  supposed  to,  the  project  manager  would  openly,  yet  jocularly,  ridicule  the  team 
member  in  front  of  his  or  her  peers.  Rather  than  using  harsh  remarks  or  other  fonns  of 
punishment,  this  technique  made  the  team  comfortable  with  each  other.  They  were  able 
to  laugh  at  their  mistakes  and  openly  discuss  project  problems. 

Gantt  charts  were  used  in  monitoring  and  managing  the  project.  The  project 
manager  felt  that  PERT  charts  were  too  complicated  and  would  lead  to  spending  too 
much  time  on  reports  instead  of  working  on  the  project.  He  also  thought  that  PERT  charts 
may  have  had  a  demoralizing  effect  on  team  members.  Therefore,  monitoring  of  the 
project  was  done  via  periodic  updates  by  team  members  and  reflected  on  Gantt  charts.  In 
project  resource  and  schedule  estimations,  a  bottom-up  approach  is  used.  The  developers 
and  team  members  were  asked  for  estimations. 

This  project  was  so  successful  that  the  system  developed  was  still  in  use  in  2000. 
The  study  participant  rated  the  overall  success  of  the  project  with  a  “10.”  The  PME  score 
for  this  project  was  “8,”  which  is  the  highest  score  in  this  data  set. 


180 


Timeframe  of  the  Proiect  1 

1980-1983 

Location 

TJ.S.A. 

Actual  Cost/  Cost  Overrun 

SI  Billion /N/A 

Actual  Schedule/Schedule  Overrun 

36  Months  /  On  Time 

Functionalitv  Delivered 

1 50% 

Size  of  the  Product 

N/A 

Average  Number  of  Peonle  involved  in  the  Proiect 

300 

Number  of  Peonle  Involved  in  Reauirements  Phase 

25 

Number  of  Peonle  involved  in  Design  Phase 

100 

Number  of  Peonle  Involved  in  imnlementation  Phase 

300 

Number  of  Peonle  Involved  in  Testing  and  Deliverv  Phase 

300 

Communication  Area  Score 

7.8 

Teamwork  Area  Score 

8.0 

Leadersh in  Area  Score 

6.9 

Organizational  Commitment  Area  Score 

7.5 

Proiect  Manager  Area  Score 

9.1 

Stakeholder  Involvement  Area  Score 

8.9 

Staffing  and  FTiring  Area  Score 

7.2 

Reauirements  Management  Area  Score 

7.3 

Proiect  Planning  and  Estimation  Area  Score 

7.9 

Proiect  Monitoring  and  Control  Area  Score 

7.9 

Scone  Management  Area  Score 

6.9 

Configuration  Management  Area  Score 

8.5 

Oualitv  Engineering  Area  Score 

8.1 

Risk  Assessment  Area  Score 

7.6 

Risk  Control  Area  Score 

8.1 

Peonle  Area  Score 

7.9 

Process  Area  Score 

7.5 

Product  Area  Score 

8.3 

Risk  Area  Score 

7.9 

PME  Score 

7.9 

Rounded  PME  Score 

8 

Proiect’s  Overall  Success  Rating 

10 

Table  26.  Project  I  Data  in  Tabular  Fonn 
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10.  Project  J 


The  goal  in  this  project  was  to  create  an  online,  real-time,  credit  card 
authorization  switching  system.  The  system  was  to  enable  merchants  who  accept  credit 
cards  to  get  quick  authorizations  from  the  banks  issuing  the  credit  cards.  This  was  a 
mainframe  software  and  hardware  project.  The  study  participant  was  the  project  manager 
of  this  project. 

This  project  was  developed  between  1976  and  1977.  The  projected  schedule  was 
twelve  months  and  the  project  took  twelve  months  to  complete.  The  projected  budget  was 
$3.7  million  while  the  actual  cost  was  only  $3.5  million.  All  of  the  functionality  from  the 
initial  baseline  was  delivered  at  the  end  of  the  project.  The  average  number  of  people 
involved  in  this  project  was  seventeen.  This  project  was  developed  by  a  major  software 
development  company  in  the  U.S. 

The  study  participant  explicitly  indicated  that  this  was  a  very  successful  project. 
When  the  study  participant  was  asked  about  the  size  of  the  product  in  terms  of  lines  of 
code,  the  participant  indicated  that  he  did  not  have  a  clue.  In  this  project,  a  substantial 
amount  of  work  went  into  the  technology,  not  the  application  code.  Therefore,  projecting 
the  size  of  the  project  from  the  code  size  would  be  quite  misleading  in  this  particular 
case.  For  example,  in  this  project  the  project  team  took  an  airline  control  program,  used 
by  the  largest  airlines  in  their  reservation  systems,  and  down-scaled  it  to  run  on  a  much  a 
smaller  mainframe.  Because  this  mini  airline  control  program  needed  a  dedicated 
environment  for  testing,  they  had  to  set  up  virtual  machines  using  a  particular  operating 
system.  The  actual  application  code  was  comparatively  simple.  The  study  participant 
indicated  that  all  people  involved  in  this  project,  both  the  developers  and  the  customer, 
were  very  motivated  to  succeed.  Everyone  was  dedicated  and  worked  whatever  hours 
were  required  to  meet  deadlines.  There  was  none  of  the  personal  competition  or 
positioning  that  seems  to  exist  in  some  projects.  The  will  to  succeed  and  the  fear  of 
failure  overwhelmed  any  personal  motivations. 
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The  author  observed  that  when  a  project  was  developed  in  the  past  and  its 
successful  operation  was  proven  throughout  the  years,  the  participants  would  declare  the 
success  of  the  project  openly  with  pride. 

The  study  participant  rated  the  overall  success  of  the  project  with  a  “10.”  The 


PME  score  for  this  project  was  “7.” 


Timeframe  of  the  Proiect  .1 

1976  -  1977 

Location 

USA. 

Actual  Cost/  Cost  Overnm 

$4.7  Million  /  None 

Actual  Schedule/Schedule  Overrun 

1 2  Months  /  On  Time 

Functionalitv  Delivered 

1 00% 

Size  of  the  Product 

N/A 

Average  Number  of  Peonle  Involved  in  the  Proiect 

17 

Number  of  Peonle  Involved  in  Reauirements  Phase 

5 

Number  of  Peonle  Involved  in  Design  Phase 

.3 

Number  of  Peonle  Involved  in  Tmnlementation  Phase 

20 

Number  of  Peonle  Involved  in  Testing  and  Deliverv  Phase 

15 

Communication  Area  Score 

8.8 

Teamwork  Area  Score 

7.8 

T.eadershin  Area  Score 

7.9 

Organizational  Commitment  Area  Score 

8.0 

Proiect  Manager  Area  Score 

8.6 

Stakeholder  Involvement  Area  Score 

6.8 

Staffing  and  Hiring  Area  Score 

6.8 

Reauirements  Management  Area  Score 

6.8 

Proiect  Planning  and  estimation  Area  Score 

7.4 

Proiect  Monitoring  and  Control  Area  Score 

7.0 

Scone  Management  Area  Score 

6.8 

Configuration  Management  Area  Score 

.6.5 

Oualitv  engineering  Area  Score 

8.4 

Risk  Assessment  Area  Score 

4.9 

Risk  Control  Area  Score 

6.1 

Peonle  Area  Score 

7.8 

Process  Area  Score 

6.7 

Product  Area  Score 

6.9 

Risk  Area  Score 

5.5 

PMB  Score _ 

_ C9 _ 
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Rounded  PMF,  Score 

7 

Project’s  Overall  Success  Rating _ 

_ 10 _ 

Table  27.  Project  J  Data  in  Tabular  Form 


11.  Project  K 

In  this  survey  study,  the  participant  was  quite  concerned  about  the  privacy  of  the 
organization.  Therefore,  the  participant  did  not  report  the  goal  and  the  deliverables  of  the 
project.  In  addition,  the  participant  did  not  report  the  projected  budget  and  actual  cost  of 
the  project.  This  data  might  lead  to  identification  of  the  organization  and  the  participant 
thought  that  this  specific  data  might  have  implications  on  the  organization’s  competitive 
advantage.  The  study  participant  was  the  project  manager  of  this  project. 

This  project  was  conducted  between  2006  and  2008.  The  projected  schedule  was 
twenty-four  months  and  the  project  took  twenty-four  months.  There  was  no  schedule  slip 
at  the  end.  The  projected  effort  for  the  project  was  172  man-months  while  the  actual 
effort  was  174  man-months.  100%  of  the  functionality  in  the  initial  baseline  was 
delivered  at  the  end  of  the  project.  The  average  number  of  people  involved  in  the 
development  was  seven.  The  product  size  was  215,400  lines  of  code.  This  project  was 
developed  by  a  company  in  Turkey. 

An  interview  could  not  be  conducted  with  the  study  participant  after  the 
completion  of  SPMEI. 

The  study  participant  rated  the  overall  success  of  the  project  with  a  “9.”  The  PME 
score  for  this  project  was  “7.” 
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Timeframe  of  the  Proiect  K 

2006-2008 

Location 

Turkev 

Actual  Cost/  Cost  Overrun 

N/A 

Actual  Schedule/Schedule  Overrun 

24  Months  /  On  Time 

Functionalitv  Delivered 

100% 

Size  of  the  Product 

215.4  KLOC 

Average  Number  of  Peonle  involved  in  the  Proiect 

7 

Number  of  Peonle  Involved  in  Reauirements  Phase 

5 

Number  of  Peonle  Involved  in  Design  Phase 

7 

Number  of  Peonle  Involved  in  tmnlementation  Phase 

8 

Number  of  Peonle  Involved  in  Testing  and  Deliverv  Phase 

7 

Communication  Area  Score 

6.8 

Teamwork  Area  Score 

6.9 

Leadersh in  Area  Score 

7.4 

Organizational  Commitment  Area  Score 

6.9 

Proiect  Manager  Area  Score 

7.4 

Stakeholder  Involvement  Area  Score 

6.0 

Staffing  and  FTiring  Area  Score 

6.5 

Reauirements  Management  Area  Score 

7.2 

Proiect  Planning  and  Estimation  Area  Score 

6.6 

Proiect  Monitoring  and  Control  Area  Score 

6.2 

Scone  Management  Area  Score 

6.3 

Configuration  Management  Area  Score 

8.7 

Oualitv  Engineering  Area  Score 

6.9 

Risk  Assessment  Area  Score 

6.4 

Risk  Control  Area  Score 

4.6 

Peonle  Area  Score 

6.8 

Process  Area  Score 

6.6 

Product  Area  Score 

7.8 

Risk  Area  Score 

5.5 

PME  Score 

6.7 

Rounded  PME  Score 

7 

Proiect’s  Overall  Success  Rating 

9 

Table  28.  Project  K  Data  in  Tabular  Fonn 
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12.  Project  L 


The  goal  in  this  project  was  to  develop  various  subsystems  of  a  large-scale  real¬ 
time  defense  system.  The  deliverables  were  the  subsystems  developed  with  all  produced 
life-cycle  documents.  This  project  was  a  subcontracted  portion  of  a  large-scale  system. 
The  study  participant  was  the  project  manager  of  this  development  effort. 

This  project  was  conducted  between  2003  and  2005.  The  projected  schedule  was 
twenty-four  months;  however,  the  project  took  thirty  months  to  complete.  There  was  a 
25%  schedule  overrun.  The  projected  effort  for  this  development  effort  was  1,024  man- 
months  while  the  actual  effort  was  much  less  than  700  man-months.  The  budget  and  the 
cost  of  the  project  will  not  be  reported  here  at  the  request  of  the  study  participant. 
However,  it  is  possible  to  report  that  the  project  was  completed  under  budget.  All  of  the 
functionality  requested  was  delivered  at  the  end  of  the  project.  Eight-hundred  people  had 
worked  on  the  development  of  this  large-scale  project.  The  average  number  of  people 
involved  in  this  subcontracted  portion  of  the  development  was  twenty-five.  The  product 
size  at  the  end  of  this  development  effort  was  440,000  lines  of  code.  This  project  was 
developed  by  a  company  in  Turkey. 

An  interview  could  not  be  conducted  with  the  study  participant  after  the 
completion  of  SPMEI. 

The  study  participant  rated  the  overall  success  of  the  project  with  a  “7.”  The  PME 
score  for  this  project  was  “6.” 
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Timeframe  of  the  Proiect  L 

2003-2005 

Location 

Turkev 

Actual  Cost/  Cost  Overrun 

N/A 

Actual  Schedule/Schedule  Overrun 

30  Months  /  25% 

Functionalitv  Delivered 

100% 

Size  of  the  Product 

440  KLOC 

Average  Number  of  Peonle  involved  in  the  Proiect 

25 

Number  of  Peonle  Involved  in  Reauirements  Phase 

100 

Number  of  Peonle  involved  in  Design  Phase 

250 

Number  of  Peonle  Involved  in  imnlementation  Phase 

350 

Number  of  Peonle  Involved  in  Testing  and  Deliverv  Phase 

100 

Communication  Area  Score 

5.1 

Teamwork  Area  Score 

5.7 

Leadersh in  Area  Score 

6.6 

Organizational  Commitment  Area  Score 

6.5 

Proiect  Manager  Area  Score 

6.3 

Stakeholder  Involvement  Area  Score 

4.2 

Staffing  and  FTiring  Area  Score 

5.7 

Reauirements  Management  Area  Score 

6.1 

Proiect  Planning  and  Estimation  Area  Score 

5.2 

Proiect  Monitoring  and  Control  Area  Score 

5.6 

Scone  Management  Area  Score 

.3.4 

Configuration  Management  Area  Score 

8.0 

Oualitv  Engineering  Area  Score 

6.2 

Risk  Assessment  Area  Score 

.3.8 

Risk  Control  Area  Score 

3.7 

Peonle  Area  Score 

5.7 

Process  Area  Score 

5.1 

Product  Area  Score 

7.1 

Risk  Area  Score 

3.7 

PME  Score 

5.5 

Rounded  PME  Score 

6 

Proiect’s  Overall  Success  Rating 

7 

Table  29.  Project  L  Data  in  Tabular  Form 
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13.  Project  M 


The  goal  of  this  project  was  to  develop  a  prototype  system  for  secure  data  transfer 
for  a  communication  system.  This  project  was  a  mix  of  software  and  hardware 
development.  The  deliverables  included  the  prototype  and  all  the  documents  produced 
during  the  life  cycle  of  the  project.  The  participant  in  this  study  was  the  project  manager 
of  this  development  effort. 

This  project  was  developed  in  2006.  The  projected  schedule  was  six  months  and 
the  actual  schedule  was  seven  months.  The  projected  effort  in  this  project  was  forty-eight 
man-months,  though  the  actual  effort  became  fifty  man-months.  Even  though  the  planned 
budget  and  actual  cost  were  reported,  they  will  not  be  stated  here  for  privacy  reasons. 
However,  it  is  possible  to  state  that  the  project  had  a  slight  overrun  in  the  budget.  95%  of 
the  functionality  from  the  initial  baseline  was  delivered  at  the  end  of  the  project.  The 
average  number  of  people  involved  in  the  development  effort  was  four.  This  project  was 
developed  by  a  government  organization  in  Turkey. 

A  detailed  interview  was  conducted  with  the  project  manager  of  this  project.  After 
the  completion  of  this  project,  the  developed  prototype  was  accepted  by  the  organization 
for  full-scale  production.  Therefore,  the  project  may  be  considered  as  a  success.  The 
project  manager  indicated  that  the  organization  was  new  to  the  domain  of  the  problem  to 
be  solved  in  this  project.  Thus,  there  were  some  risks  involved  and  the  organization 
started  with  a  prototype.  The  executive  management  was  aware  of  these  risks  and  there 
was  significant  support  for  the  successful  completion  of  the  project.  In  the  early  phases  of 
development,  the  project  team  conducted  detailed  analysis  on  the  problem  and  on  the 
solution.  The  risk  items  were  identified  early  and  incorporated  into  a  risk  management 
plan.  These  project  risks  were  avoided  whenever  possible  or  risk  mitigation  techniques 
were  used  to  reduce  their  severity.  The  project  manager  mentioned  that  he  advocates  the 
use  of  small  teams  with  talented  and  skillful  developers  when  the  environment  allows. 
Building  small  but  effective  teams  is  an  approach  used  by  many  accomplished  project 
managers.  The  small  team  size  with  skillful  developers  increased  the  communication 
effectiveness  in  this  project.  He  pointed  out  that  the  level  of  teamwork  achieved  in  this 
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project  was  high  and  the  project  environment  was  friendly.  The  project  manager  also 
indicated  that  developers  participated  in  the  project  decisions.  In  addition  to  these 
positive  features,  the  executive  management  was  very  supportive.  The  organization  was 
committed  to  the  success  of  the  project.  These  were  among  the  main  factors  in  reaching 
the  successful  outcome.  Naturally,  there  were  challenges  during  project  development. 
This  project  required  cooperation  from  various  organizations.  As  expected,  achieving 
cooperation  was  not  an  easy  task.  However,  they  were  able  to  overcome  these  challenges 
with  the  support  of  executive  management. 

The  study  participant  rated  the  overall  success  of  the  project  with  an  “8.”  The 
PME  score  for  this  project  was  “6.” 
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Timeframe  of  the  Proiect  M 

2006 

Location 

Turkev 

Actual  Cost/  Cost  Overrun 

N/A 

Actual  Schedule/Schedule  Overrun 

7  Months  /  1 7% 

Functionalitv  Delivered 

95% 

Size  of  the  Product 

115KLOC 

Average  Number  of  Peonle  involved  in  the  Proiect 

4 

Number  of  Peonle  Involved  in  Reauirements  Phase 

4 

Number  of  Peonle  Involved  in  Design  Phase 

4 

Number  of  Peonle  Involved  in  tmnlementation  Phase 

3 

Number  of  Peonle  Involved  in  Testing  and  Deliverv  Phase 

4 

Communication  Area  Score 

6.3 

Teamwork  Area  Score 

7.6 

Leadersh in  Area  Score 

8.1 

Organizational  Commitment  Area  Score 

7.4 

Proiect  Manager  Area  Score 

7.8 

Stakeholder  Involvement  Area  Score 

6.3 

Staffing  and  FTiring  Area  Score 

6.4 

Reauirements  Management  Area  Score 

6.1 

Proiect  Planning  and  Estimation  Area  Score 

5.9 

Proiect  Monitoring  and  Control  Area  Score 

6.4 

Scone  Management  Area  Score 

5.8 

Configuration  Management  Area  Score 

5.0 

Oualitv  Engineering  Area  Score 

5.4 

Risk  Assessment  Area  Score 

6.2 

Risk  Control  Area  Score 

5.0 

Peonle  Area  Score 

7.1 

Process  Area  Score 

6.0 

Product  Area  Score 

5.2 

Risk  Area  Score 

5.6 

PME  Score 

6.1 

Rounded  PME  Score 

6 

Proiect’s  Overall  Success  Rating 

8 

Table  30.  Project  M  Data  in  Tabular  Fonn 


190 


14.  Project  N 


The  goal  of  this  project  was  to  develop  $16  million  worth  of  software  upgrade  to 
a  bomber  airplane.  The  upgrade  was  to  be  conducted  on  various  functions  of  the  bomber 
plane.  Naturally,  this  project  was  a  defense  project.  The  participant  in  this  study  was  the 
program  manager  of  the  project. 

This  project  was  developed  between  2000  and  2002.  The  projected  schedule  was 
twenty-four  months  and  the  actual  schedule  was  twenty-four  months.  The  planned  budget 
and  the  actual  cost  of  the  project  were  $16  million.  So,  this  project  did  not  suffer  from  a 
schedule  or  a  cost  overrun.  However,  only  98%  of  the  functionality  from  the  initial 
baseline  was  delivered.  The  average  number  of  people  involved  in  this  development 
effort  was  nine.  This  project  was  developed  as  a  government  project  in  the  U.S.  It 
included  developers  from  the  government  agency  and  contractors  from  the  commercial 
world. 

A  brief  interview  was  conducted  with  the  study  participant.  This  project  was 
challenged  in  a  couple  ways.  During  the  development  effort,  the  project  manger  changed 
more  than  once.  The  study  participant  indicated  that  when  she  took  over,  the  project  was 
behind  schedule  and  probably  heading  for  a  cost  overrun.  She  pressured  the  contractors 
and  in-house  developer  for  better  performance.  The  strategy  she  used  was  “manage  by 
walking  around”  to  get  the  project  back  on  schedule.  She  was  among  the  developers 
almost  all  the  time.  This  is  a  strategy  sometimes  employed  by  various  project  managers. 
When  the  conditions  are  right,  this  strategy  helps  boost  performance  as  it  is  observed  in 
this  case. 

The  study  participant  rated  the  overall  success  of  the  project  with  a  “9.”  The  PME 
score  for  this  project  was  “8.” 
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Timeframe  of  the  Proiect  N 

2000-2002 

Location 

TJ.S.A. 

Actual  Cost/  Cost  Overrun 

S16  Million  /None 

Actual  Schedule/Schedule  Overrun 

24  Months  /  On  Time 

Functionalitv  Delivered 

98% 

Size  of  the  Product 

N/A 

Average  Number  of  Peonle  involved  in  the  Proiect 

9 

Number  of  Peonle  Involved  in  Reauirements  Phase 

5 

Number  of  Peonle  Involved  in  Design  Phase 

15 

Number  of  Peonle  Involved  in  tmnlementation  Phase 

10 

Number  of  Peonle  Involved  in  Testing  and  Deliverv  Phase 

8 

Communication  Area  Score 

9.2 

Teamwork  Area  Score 

7.6 

Leadersh in  Area  Score 

9.1 

Organizational  Commitment  Area  Score 

6.6 

Proiect  Manager  Area  Score 

9.2 

Stakeholder  Involvement  Area  Score 

7.5 

Staffing  and  FTiring  Area  Score 

5.8 

Reauirements  Management  Area  Score 

8.0 

Proiect  Planning  and  Estimation  Area  Score 

7.3 

Proiect  Monitoring  and  Control  Area  Score 

8.1 

Scone  Management  Area  Score 

7.7 

Configuration  Management  Area  Score 

7.2 

Oualitv  Engineering  Area  Score 

7.3 

Risk  Assessment  Area  Score 

8.1 

Risk  Control  Area  Score 

8.0 

Peonle  Area  Score 

7.9 

Process  Area  Score 

7.8 

Product  Area  Score 

7.2 

Risk  Area  Score 

8.0 

PME  Score 

7.8 

Rounded  PME  Score 

8 

Proiect’s  Overall  Success  Rating 

9 

Table  3 1 .  Project  N  Data  in  Tabular  Fonn 
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15.  Project  O 


The  goal  of  this  project  was  to  develop  a  web  application  to  collect  and  analyze 
security  related  data  over  a  secure  network.  The  participant  in  this  study  was  the 
development  lead  of  the  project. 

This  project  was  developed  between  2005  and  2007.  The  projected  schedule  was 
twenty-four  months  and  actually  took  thirty  months  to  complete.  The  budget  and  the 
actual  cost  of  the  project  were  not  reported.  This  is  either  because  of  privacy  concerns  or 
because  the  study  participant  does  not  have  access  to  this  specific  information  since  the 
data  related  to  the  cost  of  the  project  is  governed  by  a  different  group  in  the  government 
agency.  In  some  government  projects,  cost  is  not  a  priority.  The  priority  is  acquiring  the 
necessary  system  or  the  functionality.  80%  of  the  functionality  from  the  initial  baseline 
was  delivered  at  the  end  of  the  project.  The  average  number  of  people  involved  in  the 
development  effort  was  around  six  to  ten.  The  study  participant  did  not  provide  this  data 
either.  Instead,  the  participant  indicated  that  overall  twenty-nine  people  were  involved  in 
various  aspects  of  the  project.  This  project  was  developed  by  a  government  agency  in  the 
U.S.  Developers  from  the  government  agency  and  contractors  worked  on  the  project. 

An  interview  could  not  be  conducted  with  the  study  participant. 

The  study  participant  rated  the  overall  success  of  the  project  with  a  “7.”  The  PME 
score  for  this  project  was  “7.” 
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Timeframe  of  the  Proiect  O 

2005-2007 

Location 

TJ.S.A. 

Actual  Cost/  Cost  Overrun 

N/A 

Actual  Schedule/Schedule  Overrun 

30  Months  /  25% 

Functionalitv  Delivered 

80% 

Size  of  the  Product 

N/A 

Average  Number  of  Peonle  involved  in  the  Proiect 

N/A 

Number  of  Peonle  Involved  in  Reauirements  Phase 

6 

Number  of  Peonle  Involved  in  Design  Phase 

6 

Number  of  Peonle  Involved  in  tmnlementation  Phase 

12 

Number  of  Peonle  Involved  in  Testing  and  Deliverv  Phase 

5 

Communication  Area  Score 

6.3 

Teamwork  Area  Score 

6.5 

Leadersh in  Area  Score 

6.3 

Organizational  Commitment  Area  Score 

7.9 

Proiect  Manager  Area  Score 

8.1 

Stakeholder  Involvement  Area  Score 

6.5 

Staffing  and  FTiring  Area  Score 

7.9 

Reauirements  Management  Area  Score 

9.2 

Proiect  Planning  and  Estimation  Area  Score 

7.1 

Proiect  Monitoring  and  Control  Area  Score 

5.8 

Scone  Management  Area  Score 

5.6 

Configuration  Management  Area  Score 

8.2 

Oualitv  Engineering  Area  Score 

6.8 

Risk  Assessment  Area  Score 

6.2 

Risk  Control  Area  Score 

5.0 

Peonle  Area  Score 

7.1 

Process  Area  Score 

6.9 

Product  Area  Score 

7.5 

Risk  Area  Score 

5.6 

PME  Score 

6.9 

Rounded  PME  Score 

7 

Proiect’s  Overall  Success  Rating 

7 

Table  32.  Project  O  Data  in  Tabular  Fonn 
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16.  Project  P 


The  goal  of  this  project  was  to  rewrite  a  post-flight  data  analysis  system.  The 
system  includes  an  ORACLE  database,  a  robotic  storage  subsystem  and  various 
applications.  Some  of  these  FORTRAN  applications  were  rewritten  in  C.  The  study 
participant  in  this  study  was  the  lead  post-flight  analyst  of  the  development. 

The  project  took  place  between  1994  and  1995.  The  projected  schedule  was  not 
reported  by  the  study  participant,  but  the  actual  schedule  of  the  project  was  fourteen 
months.  The  budget  and  the  actual  cost  of  the  project  were  not  reported.  This  is  probably 
due  to  the  fact  that  the  study  participant  did  not  have  access  to  such  data.  This  project  was 
developed  by  a  government  agency  and  sometimes  in  these  types  of  projects,  the  cost 
data  is  kept  by  another  branch  in  the  agency.  In  addition,  acquiring  the  necessary 
functionality  has  a  higher  priority  than  the  cost  of  the  project.  All  of  the  functionality  in 
the  baseline  was  delivered  at  the  end  of  the  project.  The  average  number  of  people 
involved  in  this  project  was  ten.  This  project  was  developed  by  a  government  agency  in 
the  U.S. 

An  interview  could  not  be  conducted  with  the  study  participant. 

The  study  participant  rated  the  overall  success  of  the  project  as  a  “9.”  The  PME 
score  for  this  project  was  “9.” 


Timeframe  of  the  Project  P 


1994-1995 
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Location 

TJ.S.A. 

Actual  Cost/  Cost  Overrun 

N/A 

Actual  Schedule/Schedule  Overrun 

14  Months  /  17% 

Functionalitv  Delivered 

100% 

Size  of  the  Product 

N/A 

Average  Number  of  Peonle  involved  in  the  Proiect 

10 

Number  of  Peonle  Involved  in  Reauirements  Phase 

10 

Number  of  Peonle  Involved  in  Design  Phase 

10 

Number  of  Peonle  involved  in  Imnlementation  Phase 

5 

Number  of  Peonle  Involved  in  Testing  and  Deliverv  Phase 

5 

Communication  Area  Score 

9.0 

Teamwork  Area  Score 

9.6 

Leadershin  Area  Score 

9.1 

Organizational  Commitment  Area  Score 

10.0 

Proiect  Manager  Area  Score 

9.6 

Stakeholder  Involvement  Area  Score 

8.3 

Staffing  and  FTiring  Area  Score 

9.8 

Reauirements  Management  Area  Score 

9.7 

Proiect  Planning  and  Estimation  Area  Score 

8.7 

Proiect  Monitoring  and  Control  Area  Score 

8.1 

Scone  Management  Area  Score 

7.9 

Configuration  Management  Area  Score 

9.3 

Oualitv  Engineering  Area  Score 

9.7 

Risk  Assessment  Area  Score 

8.5 

Risk  Control  Area  Score 

5.6 

Peonle  Area  Score 

9.4 

Process  Area  Score 

8.6 

Product  Area  Score 

9.5 

Risk  Area  Score 

7.0 

PME  Score 

8.8 

Rounded  PME  Score 

9 

Proiect’s  Overall  Success  Rating 

9 

Table  33.  Project  P  Data  in  Tabular  Form 
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17.  Project  R 


In  this  survey  study,  the  study  participant  and  his  executive  manager  were  very 
much  concerned  with  the  privacy  of  the  organization.  Therefore,  they  did  not  report  many 
of  the  overall  project  statistics  such  as  the  budget,  cost,  schedule,  number  of  people 
involved,  etc.  In  fact,  the  executive  manager  said  that  this  type  of  infonnation  was  only 
accessible  to  certain  people  in  the  organization.  Their  company  policy  strictly  prohibits 
releasing  such  information  to  third  parties.  Due  to  lack  of  data,  this  project  could  not  be 
categorized.  So,  this  survey  study  was  excluded  from  the  main  dataset.  However,  there  is 
an  important  piece  of  information  gathered  as  the  result  of  this  study.  This  project  was 
developed  by  an  organization  with  a  CMMI  level  5  rating.  Consequently,  it  is  expected 
that  the  project  management  effectiveness  in  this  software  project  was  high.  The  results 
of  the  study  confinned  what  was  expected  —  this  project  reached  the  highest  project 
management  effectiveness  score  among  the  projects  analyzed  in  this  research.  The  PME 
score  for  this  project  was  “9.” 

The  goal  in  this  project  was  to  develop  an  electronic  warfare  training  simulation 
system  to  train  military  personnel  for  electronic  warfare  concepts  and  operations.  The 
study  participant  was  the  project  manager  of  this  project.  The  communication  for  the 
participation  to  the  study  was  first  directed  to  the  public  relations  officer  of  the  company. 
The  public  relations  officer  passed  the  request  to  the  executive  manager  of  the  project 
manager.  The  executive  manager  analyzed  the  request  and  agreed  to  participation  in  the 
study  and  after  that,  the  executive  manager  acted  as  an  intermediary  during  the  study.  The 
professionalism  in  this  process  was  an  indication  of  the  established  corporate  culture  of 
this  organization. 

A  detailed  interview  was  conducted  with  the  study  participant.  The  participant 
indicated  that  this  project  was  his  first  project  as  a  project  manager  in  this  organization. 
He  also  mentioned  that  he  worked  in  other  organizations  prior  to  joining  his  current 
organization.  The  participant  pointed  out  that  the  commitment  to  excellence  in  this 
organization  especially  by  the  executive  management  was  notable.  The  organization  has  a 
CMMI  level  5  rating  and  other  quality  certifications.  In  addition,  the  executive 
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management  was  committed  to  excellence  and  quality  in  every  aspect  of  software  project 
development.  The  study  participant  said  that  the  project  was  overseen  by  an  oversight 
committee.  This  oversight  committee  was  internal  to  the  organization  but  external  to  the 
project  team.  The  task  of  the  committee  was  to  ensure  that  necessary  and  adequate 
processes  were  in  place  to  achieve  high  quality  in  project  management  and  software 
development.  This  committee  would  not  allow  the  projects  to  derail. 

In  this  project,  the  project  team  consisted  of  talented  and  experienced  developers. 
However,  the  developers  were  not  familiar  with  the  domain.  Therefore,  there  was  a 
learning  curve  during  the  development.  The  productivity  was  low  compared  to  other 
projects  at  the  beginning,  but  when  the  developers  became  familiar  with  the  domain,  the 
productivity  quickly  boosted.  Another  challenging  part  of  this  project  was  that  the  project 
required  increased  coordination  among  various  stakeholders.  There  were  more  than  a 
couple  of  organizations  on  the  client  side.  Logistics  such  as  traveling  and  arrangement  of 
meetings  was  an  issue  in  this  project.  The  support  from  executive  management  helped  to 
overcome  these  challenges. 

The  study  participant  indicated  that  the  coverage  of  SPMEI  was  good.  He 
mentioned  that  SPMEI  covered  most  essential  project  management  areas.  Since  the 
organization  has  recognized  quality  certifications,  strict  quality  controls  and  good 
software  development  processes  and  procedures  were  already  in  place.  Most  of  these 
were  covered  in  SPMEI,  so  it  was  easy  for  him  to  respond  to  SPMEI.  In  addition,  he 
mentioned  that  SPMEI  pointed  out  some  practices  they  were  not  employing  at  the  time. 
Some  of  these  are  people  management  related  practices.  According  to  him,  these  were 
good  to  know. 

The  study  participant  rated  the  overall  success  of  the  project  with  a  “7.”  The  PME 
score  for  this  project  was  “9.” 
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Timeframe  of  the  Proiect  R 

N/A 

Location 

Eurone 

Actual  Cost/  Cost  Overrun 

N/A 

Actual  Schedule/Schedule  Overrun 

N/A 

Functionalitv  Delivered 

N/A 

Size  of  the  Product 

N/A 

Average  Number  of  Peonle  involved  in  the  Proiect 

N/A 

Number  of  Peonle  Involved  in  Reauirements  Phase 

N/A 

Number  of  Peonle  involved  in  Design  Phase 

N/A 

Number  of  Peonle  Involved  in  imnlementation  Phase 

N/A 

Number  of  Peonle  Involved  in  Testing  and  Deliverv  Phase 

N/A 

Communication  Area  Score 

9.5 

Teamwork  Area  Score 

8.4 

Leadersh in  Area  Score 

7.6 

Organizational  Commitment  Area  Score 

8.2 

Proiect  Manager  Area  Score 

7.3 

Stakeholder  Involvement  Area  Score 

9.6 

Staffing  and  FTiring  Area  Score 

7.6 

Reauirements  Management  Area  Score 

9.2 

Proiect  Planning  and  Estimation  Area  Score 

8.4 

Proiect  Monitoring  and  Control  Area  Score 

9.3 

Scone  Management  Area  Score 

7.6 

Configuration  Management  Area  Score 

9.0 

Oualitv  Engineering  Area  Score 

9.8 

Risk  Assessment  Area  Score 

9.6 

Risk  Control  Area  Score 

6.9 

Peonle  Area  Score 

8.3 

Process  Area  Score 

8.6 

Product  Area  Score 

9.4 

Risk  Area  Score 

8.2 

PME  Score 

8.6 

Rounded  PME  Score 

9 

Proiect’s  Overall  Success  Rating 

7 

Table  34.  Project  R  Data  in  Tabular  Form 
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18.  Project  X 


The  goal  in  this  project  was  to  add  functionality  to  an  existing  system.  The  system 
was  a  hospital’s  patient  admission  system.  The  new  functionality  was  to  improve  the 
admission  system  with  enhanced  statistical  analysis  capabilities.  The  final  product 
included  the  executables,  source  code  and  a  manual.  This  application  was  not  intended  to 
be  a  commercial  product,  but  an  experimental  tool  to  be  used  by  management  scientists 
and  hospital  management.  The  study  participant  was  the  analyst  and  the  developer  of  the 
project. 

This  project  was  developed  in  2007.  The  projected  schedule  was  three  months, 
and  the  project  was  completed  in  only  two  months.  The  planned  effort  was  two  man- 
months  and  the  actual  effort  was  1.5  man-months.  The  budget  and  the  cost  of  the  project 
were  not  reported.  95%  of  the  functionality  in  the  baseline  was  delivered.  The  average 
number  of  people  involved  in  the  development  of  this  project  was  two.  Only  one 
developer  was  used  in  the  design  and  implementation  phase.  Only  three  people  were 
involved  in  the  overall  project.  The  size  of  the  product  was  4,489  lines  of  code.  This 
project  was  mainly  developed  by  a  graduate  student  in  an  academic  environment  in  the 
United  Kingdom. 

The  study  participant  rated  the  overall  success  of  the  project  with  an  “.8.  The 
PME  score  for  this  project  was  a  “4.” 

The  metric  proposed  in  this  doctoral  research  is  not  applicable  to  this  project. 
Even  though  the  metric  is  applicable  to  a  wide  variety  of  software  projects,  of  course 
there  are  exceptions.  Even  a  brief  introduction  of  the  project  presented  above  explains 
why  this  metric  is  not  applicable  in  this  specific  case.  For  the  most  part,  the  project  was 
developed  by  only  one  person.  It  is  likely  that  the  graduate  student  developed  the  project 
in  addition  to  his  other  responsibilities.  The  total  effort  was  only  1.5  man-months  and  the 
project  was  developed  in  just  two  months.  Basically,  this  was  a  small  maintenance  effort 
to  enhance  some  of  the  functionality  in  an  existing  system.  There  was  no  project  plan.  No 
project  management  data  or  metrics  gathered.  There  was  no  requirements  change  and 
control  process,  no  risk  management  process,  no  configuration  management  process,  no 
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scope  and  baseline  tracking  process.  No  project  estimation  was  used.  A  detailed  analysis 
of  the  responses  would  reveal  why  the  metric  is  not  applicable.  Therefore,  in  this  project 
expectance  of  a  full-scale  project  management  effort  and  trying  to  measure  its 
effectiveness  would  be  far  from  meaningful.  This  survey  study  is  included  here  in  order 
to  show  an  example  of  the  cases  in  which  the  metric  is  not  applicable.  It  is  likely  this 
project  was  as  a  successful  project.  It  is  just  that  the  metric  proposed  will  not  provide  any 
meaningful  measurement  in  this  case. 

There  is  also  a  more  concrete  way  to  identify  whether  the  metric  is  applicable  to  a 
particular  project  or  not.  An  analysis  of  the  responses  in  the  SPMEI  would  reveal  the 
applicability.  The  number  of  “Not  Applicable”  responses  is  an  indicator  for  applicability 
check.  In  this  survey  study,  the  participant  responded  with  a  “Not  Applicable”  to  the 
statements  and  questions  in  approximately  forty  different  instances.  This  number  is  much 
higher  than  the  number  of  “Not  Applicable”  and  “None”  responses  observed  in  the  cases 
where  the  metric  is  applicable. 
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Timeframe  of  the  Proiect  X 

2007 

Location 

IJK 

Actual  Cost/  Cost  Overrun 

N/A 

Actual  Schedule/Schedule  Overrun 

2  Months  /  On  Time 

Functionalitv  Delivered 

95% 

Size  of  the  Product 

4.5  KLOC 

Average  Number  of  Peonle  involved  in  the  Proiect 

2 

Number  of  Peonle  Involved  in  Reauirements  Phase 

2 

Number  of  Peonle  Involved  in  Design  Phase 

1 

Number  of  Peonle  Involved  in  tmnlementation  Phase 

1 

Number  of  Peonle  Involved  in  Testing  and  Deliverv  Phase 

2 

Communication  Area  Score 

6.3 

Teamwork  Area  Score 

6.7 

Leadersh in  Area  Score 

8.5 

Organizational  Commitment  Area  Score 

6.7 

Proiect  Manager  Area  Score 

7.2 

Stakeholder  Involvement  Area  Score 

4.4 

Staffing  and  FTiring  Area  Score 

5.3 

Reauirements  Management  Area  Score 

4.6 

Proiect  Planning  and  Estimation  Area  Score 

3.0 

Proiect  Monitoring  and  Control  Area  Score 

2.8 

Scone  Management  Area  Score 

2.8 

Configuration  Management  Area  Score 

2.2 

Oualitv  Engineering  Area  Score 

3.5 

Risk  Assessment  Area  Score 

2.4 

Risk  Control  Area  Score 

2.0 

Peonle  Area  Score 

6.5 

Process  Area  Score 

3.3 

Product  Area  Score 

2.9 

Risk  Area  Score 

2.2 

PME  Score 

4.1 

Rounded  PME  Score 

4 

Proiect’s  Overall  Success  Rating 

8 

Table  35.  Project  X  Data  in  Tabular  Fonn 
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19.  Project  Y 


The  goal  in  this  project  was  to  create  a  Java  batch  application  to  process  requests 
for  printed  letters  to  be  generated.  The  deliverable  was  a  JAR  (Java  Archive)  file. 

This  project  was  conducted  in  2008.  The  projected  schedule  was  three  months,  yet 
the  project  was  completed  in  only  two  months.  The  planned  effort  was  3.25  man-months 
and  the  actual  effort  was  2.25  man-months.  The  budget  and  the  cost  of  the  project  were 
not  known.  All  of  the  functionality  was  delivered  at  the  end.  The  product  size  was  only 
1,500  lines  of  code.  The  total  number  of  people  involved  in  this  effort  was  four.  The 
calculated  average  number  of  people  was  2.25.  This  project  was  developed  in  the  U.S.  as 
a  government  contract. 

The  study  participant  rated  the  overall  success  of  the  project  with  a  “10.”  The 
PME  score  for  this  project  was  a  “4.” 

This  study  is  another  example  for  a  case  in  which  the  metric  is  not  applicable.  The 
final  product  in  this  project  was  a  very  small  application.  It  is  a  batch  application  that 
processes  outputs  from  other  applications.  This  project  may  be  viewed  as  a  maintenance 
effort  because  of  the  application  developed.  The  total  effort  was  small,  and  the  number  of 
people  involved  in  this  effort  was  few.  Most  importantly,  the  project  scope  was  very 
limited.  In  every  aspect,  this  project  is  a  small  project.  In  these  types  of  projects,  there  is 
no  need  for  a  full-blown  project  management  effort.  It  may  even  get  in  the  way  of 
developers  involved  due  to  unnecessary  overhead.  Because  the  scope  is  small  and  the 
schedule  is  very  short,  it  is  even  possible  to  start  over  if  the  project  fails  as  the 
consequences  of  starting  over  would  not  be  catastrophic  compared  to  bigger  projects. 

Again,  this  survey  study  is  included  to  show  an  example  of  the  cases  in  which  the 
metric  is  not  applicable.  There  were  over  forty  “Not  Applicable”  and  “None”  responses 
in  the  completed  SPMEI. 
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Timeframe  of  the  Proiect  Y 

2008 

Location 

TJ.S.A. 

Actual  Cost/  Cost  Overrun 

N/A 

Actual  Schedule/Schedule  Overrun 

2  Months  /  On  Time 

Functionalitv  Delivered 

100% 

Size  of  the  Product 

1 .5  KLOC 

Average  Number  of  Peonle  involved  in  the  Proiect 

2 

Number  of  Peonle  Involved  in  Reauirements  Phase 

1 

Number  of  Peonle  Involved  in  Design  Phase 

2 

Number  of  Peonle  Involved  in  tmnlementation  Phase 

3 

Number  of  Peonle  Involved  in  Testing  and  Deliverv  Phase 

3 

Communication  Area  Score 

4.5 

Teamwork  Area  Score 

5.9 

Leadersh in  Area  Score 

5.1 

Organizational  Commitment  Area  Score 

5.1 

Proiect  Manager  Area  Score 

6.3 

Stakeholder  Involvement  Area  Score 

4.7 

Staffing  and  FTiring  Area  Score 

5.3 

Reauirements  Management  Area  Score 

3.5 

Proiect  Planning  and  Estimation  Area  Score 

4.1 

Proiect  Monitoring  and  Control  Area  Score 

3.5 

Scone  Management  Area  Score 

4.8 

Configuration  Management  Area  Score 

3.5 

Oualitv  Engineering  Area  Score 

4.0 

Risk  Assessment  Area  Score 

3.6 

Risk  Control  Area  Score 

4.4 

Peonle  Area  Score 

5.3 

Process  Area  Score 

4.0 

Product  Area  Score 

3.7 

Risk  Area  Score 

4.0 

PME  Score 

4.4 

Rounded  PME  Score 

4 

Proiect’s  Overall  Success  Rating 

10 

Table  36.  Project  Y  Data  in  Tabular  Fonn 


204 


20.  Project  Z 


The  goal  in  this  project  was  to  develop  an  ALGOL  compiler  for  a  supercomputer. 
The  participant  was  the  project  manager  of  this  project. 

This  project  was  developed  in  between  1974  and  1975.  The  projected  schedule 
was  nine  months,  though  the  project  took  twelve  months  to  complete.  The  planned  effort 
for  the  project  was  nine  man-months  and  the  actual  effort  was  eighteen  man-months.  The 
budget  for  the  project  was  $50,000,  while  the  cost  of  the  project  was  $80,000.  95%  of  the 
functionality  in  the  baseline  was  delivered  at  the  end.  The  average  number  of  people 
involved  in  the  development  of  this  project  was  four.  This  project  was  developed  in  the 
U.S.  in  an  academic  environment.  The  project  manager  and  the  developers  were  faculty 
and  graduate  students. 

A  brief  interview  was  conducted  with  the  project  manager  after  the  providing 
responses  to  SPMEI.  The  study  participant  indicated  that  this  project  was  a  small  project 
done  over  thirty  years  ago  when  the  participant  was  a  highly  inexperienced  project 
manager.  The  three  graduate  students  who  were  involved  in  the  development  were  also 
highly  inexperienced  with  real  projects  for  real  customers.  The  study  participant  indicated 
that  they  did  not  know  how  to  run  a  real  project  and  made  all  kinds  of  mistakes.  On  the 
other  hand,  the  requirements  in  the  project  were  relatively  fixed  which  worked  well  for 
them.  They  had  quite  a  challenge  that  contributed  to  late  delivery  and  increased  cost.  The 
project  team  could  not  gain  the  necessary  access  to  the  target  computer  for  which  they 
were  developing  software.  They  could  not  execute  the  test  runs  as  planned. 

The  study  participant  rated  the  overall  success  of  the  project  with  “8.”  The  PME 
score  for  this  project  was  “4.” 

The  proposed  metric  is  not  applicable  to  this  project  for  a  couple  of  reasons.  The 
main  reason  is  that  this  project  was  developed  in  1974.  In  those  years,  the  body  of 
knowledge  in  project  management  was  quite  limited  when  compared  with  our  knowledge 
in  that  area  today.  SPMEI  incorporates  the  state  of  art  in  project  management.  Most  of 
the  best  practices  inquired  in  a  software  project  with  SPMEI  were  not  even  known  at  the 
time.  For  example,  SPMEI  inquires  whether  earned  value  management  is  used  in  the 
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project  or  not.  When  this  project  was  being  developed,  earned  value  management  was  not 
introduced.  Another  example  is  level  of  automation  used  in  the  project  management 
inquired  in  SPMEI  in  various  project  management  areas.  In  1974,  the  project  team  did 
not  have  the  computer  aided  software  engineering  (CASE)  tools  used  today.  In  addition, 
the  development  team  was  small  and  composed  of  graduate  students  and  faculty.  It  is 
very  likely  that  these  people  had  other  responsibilities  unlike  the  people  in  various 
software  development  companies  and  organizations. 

In  addition,  a  detailed  analysis  of  the  responses  in  SPMEI  will  show  why  the 
metric  is  not  applicable.  There  was  no  project  plan  in  this  effort,  and  the  tasks  and 
activities  were  identified  as  the  project  progressed.  Project  estimation  techniques  were 
not  used  and  no  project  management  data  or  metrics  were  gathered.  There  was  no  risk 
assessment  and  the  risks  were  only  identified  as  the  project  evolved.  There  were  over 
twenty  “Not  Applicable”  and  “None”  responses.  This  number  is  clearly  higher  than  the 
projects  for  which  the  metric  is  applicable. 
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Timeframe  of  the  Proiect  7, 

1974-1975 

Location 

TJ.S.A. 

Actual  Cost/  Cost  Overrun 

S80.000  /  60% 

Actual  Schedule/Schedule  Overrun 

12  Months  /  34% 

Functionalitv  Delivered 

95% 

Size  of  the  Product 

N/A 

Average  Number  of  Peonle  Involved  in  the  Proiect 

4 

Number  of  Peonle  Involved  in  Reauirements  Phase 

4 

Number  of  Peonle  Involved  in  Design  Phase 

4 

Number  of  Peonle  Involved  in  tmnlementation  Phase 

4 

Number  of  Peonle  Involved  in  Testing  and  Deliverv  Phase 

4 

Communication  Area  Score 

4.6 

Teamwork  Area  Score 

4.9 

Leadersh in  Area  Score 

5.7 

Organizational  Commitment  Area  Score 

6.2 

Proiect  Manager  Area  Score 

5.7 

Stakeholder  Involvement  Area  Score 

3.3 

Staffing  and  FTiring  Area  Score 

4.7 

Reauirements  Management  Area  Score 

4.8 

Proiect  Planning  and  Estimation  Area  Score 

2.9 

Proiect  Monitoring  and  Control  Area  Score 

2.2 

Scone  Management  Area  Score 

.3.0 

Configuration  Management  Area  Score 

2.2 

Oualitv  Engineering  Area  Score 

2.6 

Risk  Assessment  Area  Score 

2.9 

Risk  Control  Area  Score 

2.8 

Peonle  Area  Score 

5.0 

Process  Area  Score 

3.2 

Product  Area  Score 

2.4 

Risk  Area  Score 

2.8 

PME  Score 

3.6 

Rounded  PME  Score 

4 

Proiect’s  Overall  Success  Rating 

8 

Table  37.  Project  Z  Data  in  Tabular  Form 
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D.  DATA  ANALYSIS 

In  the  previous  section,  each  project  is  presented  in  detail  one-by-one.  In  this 
section,  the  data  from  survey  studies  will  be  analyzed  as  a  dataset.  In  the  rest  of  the 
section,  “N/A”  indicates  “Not  Available.” 

A  total  of  twenty  software  projects  were  analyzed  in  this  survey  study.  Sixteen  of 
them  were  grouped  together  as  a  dataset,  which  includes  Project  A  to  Project  P.  It  is 
observed  that  the  metric  is  not  applicable  to  three  projects:  Projects  X,  Y,  and  Z.  In  one 
study,  the  participant  was  not  able  to  provide  project  data  that  would  enable  us  to 
categorize  the  project.  Therefore,  this  project,  Project  R,  was  excluded  from  the  main 
dataset. 

The  projects  in  the  dataset  are  a  good  mix  of  projects  in  many  aspects.  The  dataset 
includes  projects  from  every  decade  since  the  1970s.  In  addition,  most  of  the  projects 
were  developed  in  recent  years.  These  projects  provide  an  emphasis  on  the  practices 
employed  today  in  the  software  industry.  The  development  length  of  the  projects  in  the 
dataset  varies  from  four  months  to  three  years.  In  the  software  industry,  very  few  projects 
expand  to  more  than  three  years  of  development  time.  Most  of  the  study  participants  did 
not  or  could  not  report  the  cost  of  the  projects.  The  study  participants  had  privacy 
concerns.  The  rest  of  the  projects  show  variability  in  cost.  Almost  all  of  the  participants 
reported  the  percentage  of  delivered  functionality  compared  to  baseline.  In  most  of  the 
projects,  more  than  90%  of  the  functionality  from  the  initial  baseline  was  delivered.  Most 
of  the  projects  were  delivered  on  time.  There  were  eight  projects  with  a  schedule  overrun. 
All  of  them,  except  one,  had  less  than  50%  schedule  overrun.  The  delivered  systems  or 
products  include  real-time  systems,  embedded  systems,  mission  critical  systems, 
information  management  systems,  office  automation  systems,  etc. 

All  of  the  projects  in  the  dataset  were  developed  in  North  America  and  Europe. 
The  dataset  does  not  include  projects  developed  in  Asia,  South  America  and  Australia. 
Today,  a  big  portion  of  the  software  development  efforts  takes  place  in  North  America 
and  Europe.  However,  China,  India,  Taiwan  and  some  other  countries  have  increased 
their  share  in  the  recent  years.  Future  studies  should  include  projects  developed  in  these 
parts  of  the  world. 


208 


Table  38  provides  an  overview  of  basic  statistics  gathered  from  the  survey  studies  on  software  projects. 


Project 

Identifier 

Time 

Frame 

Projected/ 

Estimated 

Schedule 

(months) 

Actual 

Schedule 

(months) 

Schedule 

Overrun 

Projected/ 

Planned 

Effort 

Actual 

Effort 

Projected/ 

Planned 

Budget 

Actual 

Cost 

Cost 

Overrun 

Delivered 

Functionality 

Proiect  A 

2006 

7 

10 

43% 

N/A 

N/A 

S500K 

S750K 

50% 

100% 

Project  B 

2006 

6 

6 

On  Time 

N/A 

N/A 

N/A 

N/A 

N/A 

95% 

Project  C 

2007-2009 

12 

17 

42% 

125 

175 

S3. 5  M 

S4.5  M 

28.5% 

100% 

Project  D 

2006 

6 

4 

On  Time 

24 

16 

S29K 

S20K 

None 

100% 

Project  R 

1993-1995 

9 

24 

167% 

180 

360 

N/A 

N/A 

N/A 

70% 

Project  F 

2007-2008 

10 

12 

20% 

220 

275 

N/A 

N/A 

N/A 

95% 

Project  G 

1982-1983 

10 

10 

On  Time 

96 

96 

N/A 

N/A 

N/A 

70% 

Project  Fi 

2003-2005 

24 

24 

On  Time 

200 

230 

S3. 2  M 

S3. 2  M 

0% 

90% 

Project  I 

1980-1983 

36 

36 

On  Time 

1.000 

1.000 

N/A 

SI  R 

N/A 

1 50% 

Project  J 

1976-1977 

12 

12 

On  Time 

N/A 

N/A 

S4.7M 

S4.5  M 

None 

100% 

Project  K 

2006-2008 

24 

24 

On  Time 

172 

174 

N/A 

N/A 

N/A 

100% 

Project  L 

2003-2005 

24 

30 

25% 

1.024 

700 

N/A 

N/A 

N/A 

100% 

Project  M 

2006 

6 

7 

17% 

48 

50 

N/A 

N/A 

N/A 

95% 

Project  N 

2000-2002 

24 

24 

On  Time 

24 

24 

S16M 

S16M 

None 

98% 

Project  0 

2005-2007 

24 

30 

25% 

N/A 

N/A 

N/A 

N/A 

N/A 

80% 

Project  P 

1994-1995 

12 

14 

17% 

N/A 

N/A 

N/A 

N/A 

N/A 

100% 

Table  38.  Overview  of  Projects  in  the  Dataset 
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Table  39  shows  an  overview  of  data  gathered  from  the  studies  in  terms  of  the 
number  of  people  involved  in  different  phases  of  the  project  development,  as  well  as  the 
total  and  average  number  of  people  involved  in  the  development  effort.  In  most  studies, 
the  participants  did  not  provide  the  total  number  of  people  involved  in  the  project.  On  the 
other  hand,  almost  all  of  the  participants  provided  the  average  number  of  people  involved 
in  the  project.  Thus,  it  is  possible  to  categorize  the  projects  based  on  the  average  number 
of  people  involved. 


Project 

Identifier 

Requirements 

Phase 

Design 

Phase 

Implementation 

Phase 

Testing 

and 

Delivery 

Phase 

Total 

Number 

of  People 

Involved 

Average 

Number  of 

People 

Involved 

Proiect  A 

3 

2 

4 

4 

N/A 

4 

Proiect  B 

4 

10 

15 

10 

N/A 

10 

Proiect  C 

3 

3 

4 

4 

18 

10 

Proiect  D 

4 

4 

5 

5 

N/A 

5 

Proiect  E 

4 

4 

20 

15 

43 

N/A 

Proiect  F 

5 

14 

16 

22 

N/A 

20 

Proiect  G 

8 

10 

10 

10 

N/A 

10 

Proiect  H 

2 

2 

4 

5 

13 

8 

Proiect  I 

25 

100 

300 

300 

N/A 

300 

Proiect  J 

5 

3 

20 

15 

N/A 

17 

Proiect  K 

5 

7 

8 

7 

7 

7 

Proiect  L 

100 

250 

350 

100 

800 

25 

Proiect  M 

4 

4 

3 

4 

N/A 

4 

Proiect  N 

5 

15 

10 

8 

38 

9 

Proiect  0 

6 

6 

12 

5 

29 

N/A 

Proiect  P 

10 

10 

5 

5 

N/A 

10 

Table  39.  Overview  of  Projects:  Number  of  People  Involved 


Figure  37  presents  the  distribution  of  projects  in  terms  of  average  number  of 
people  involved.  The  projects  are  divided  into  four  size  categories:  small,  medium,  large 
and  very  large.  A  small  size  project  involves  4-9  people  on  average  during  the 
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development  effort.  A  medium  size  project  involves  10-19  people  on  average.  A  large 
size  project  involves  20-100  people  on  average.  A  very  large  project  involves  more  than 
100  people  on  average  during  the  development.  Almost  half  of  the  projects  in  the  dataset 
are  small  size  projects.  Close  to  one-third  of  the  projects  are  medium  sized  projects  while 
close  to  one-fourth  of  the  projects  are  large  size  projects.  There  is  one  project  with  more 
than  100  people  involved  on  average. 


Distribution  of  Projects  in  the  Dataset 
Project  Size  In  Terms  of  Average  Number  of  People 
Involved  in  the  Project 


Very  Large  Size 
Project  (100+ 
People),  1 , 6% 


Large  Size 


Project  (20-100 
People),  3,  19% 


Small  Size 
Project  (4-9 
People) ,  7, 
44% 


Medium  Size  / 

Project  (10-19 
People) ,  5, 

31% 

□  Small  Size  Project  (4-9  People) 

■  Medium  Size  Project  (10-19  People) 

□  Large  Size  Project  (20-100  People) 

□  Very  Large  Size  Project  (100+  People) 


Figure  37.  Distribution  of  Project  Size  in  Terms  of  Average  Number  of  People  Involved 
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Table  40  provides  the  size  of  the  products  delivered  at  the  end  of  the  projects. 
Even  though  more  than  half  of  the  study  participants  reported  this  data,  most  of  these 
reports  do  not  include  exact  numbers.  During  the  interviews  with  the  study  participants,  it 
was  observed  that  the  study  participants  are  not  very  concerned  about  the  final  product 
size.  They  were  more  concerned  with  whether  the  project  team  delivered  the  functionality 
or  not. 


Project 

LOC  (Number  of  Lines  of 

Number  of  Function  Points 

Identifier 

Code) 

Proiect  A 

100.000  tAnnroximatelvt 

N/A 

Project  B 

N/A 

N/A 

Project  C 

100.000  tAnnroximatelvt 

N/A 

Project  D 

30.000 

N/A 

Project  E 

N/A 

More  than  30.000 

Project  F 

N/A 

N/A 

Project  G 

16.000 

N/A 

Project  H 

10.000 

50 

Project  I 

N/A 

N/A 

Project  J 

N/A 

N/A 

Project  K 

215.400 

N/A 

Project  E 

440.000 

N/A 

Project  M 

1  15.000 

N/A 

Project  N 

N/A 

N/A 

Project  O 

N/A 

N/A 

Project  P 

N/A 

N/A 

Table  40.  Product  Size 


Figures  38  and  39  present  the  timeframe  of  the  projects  in  the  dataset.  Most  of  the 
projects  in  the  dataset  were  developed  after  2000.  Thus,  the  dataset  provides  a  focus  to 
the  project  practices  in  the  recent  years.  There  are  also  sample  projects  from  every  decade 
since  1970s.  The  number  of  samples  is  not  enough  to  identify  trends  in  the  project 
management  practices  employed  at  different  timeframes. 
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Figure  38.  Time  Frame  of  Projects  Analyzed 


Time  Frame  of  Projects  Analyzed 


1970s,  1  project 


Figure  39.  Time  Frame  of  Projects  in  the  Dataset  Categorized  by  Decades 
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The  actual  schedules  of  projects  in  the  dataset  are  presented  in  Figure  40.  This 
data  element  is  one  of  the  few  items  reported  by  every  participant  in  the  study.  The  actual 
schedules  vary  from  four  months  to  thirty-six  months.  The  average  schedule  of  the 
projects  in  the  dataset  is  eighteen  months.  Four  of  the  projects  took  twenty-four  months 
to  complete.  This  is  the  largest  category. 


Actual  Schedule  of  Projects  in  the  Dataset 

4  Months,  1 
Project 


36  Months,  1 
Project 

30  Months,  2 
Projects 


24  Months,  4 
Projects 


6  Months,  1 
Project 

7  Months,  1 
Project 


10  Months,  2 
Projects 


1 2  Months,  2 
Projects 


17  Months,  1 
Project 


14  Months,  1 
Project 


□  4  Months,  1  Project  B  6  Months,  1  Project  □  7  Months,  1  Project 

□  10  Months,  2  Projects  B  12  Months,  2  Projects  □  14  Months,  1  Project 

□  17  Months,  1  Project  □  24  Months,  4  Projects  B  30  Months,  2  Projects 
B  36  Months,  1  Project 


Figure  40.  Actual  Schedule  of  Projects  in  the  Dataset 
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Figure  4 1 .  Distribution  of  PME  Scores  -  Bar  Chart 


Figure  42.  Distribution  of  PME  Scores  -  Scatter  Plot 

Figures  41  and  42  present  the  distribution  of  PME  scores  in  the  dataset  as  a  bar 
and  scatter  chart.  From  the  figures,  it  is  possible  to  speculate  that  the  distribution  of 
scores  resembles  a  normal  distribution.  The  lack  of  knowledge  about  the  overall 
population  of  software  projects  limits  our  ability  to  test  whether  this  sample  distribution 
follows  the  statistical  distribution  of  the  overall  population  or  not.  Therefore,  until  there 
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is  evidence  that  this  sample  distribution  of  project  management  effectiveness  (PME) 
scores  of  software  projects  is  not  a  fair  representation  of  the  overall  distribution,  we  will 
assume  that  it  is. 

Distribution  of  Success  Ratings 

■  Success  Rating 


6 


0123456789  10 

Success  Rating 


Figure  43.  Distribution  of  Success  Ratings  -  Bar  Chart 

Figure  43  presents  the  distribution  of  project  success  ratings  provided  by  study 
participants  for  their  projects.  It  is  observed  that  most  of  the  projects  in  the  dataset  are 
successful  projects  as  indicated  by  the  study  participants.  However,  this  does  not 
necessarily  mean  that  this  data  is  not  a  fair  representation  of  the  population. 
Unfortunately,  what  percentage  of  software  projects  is  a  success  is  still  an  open  question 
today.  Therefore,  such  lack  of  knowledge  limits  our  ability  to  conduct  further  analysis  on 
this  set  of  data. 

Figure  44  shows  the  plot  of  project  success  scores  and  rounded  project 
management  effectiveness  (PME)  scores  for  each  project.  An  analysis  of  the  figure 
suggests  that  there  is  a  close  relationship  with  the  success  of  a  software  project  and 
project  management  effectiveness.  The  trend  acquired  by  plotting  the  project  success 
ratings  for  each  project  is  similar  to  the  trend  acquired  by  plotting  the  project 
management  effectiveness  (PME)  scores  for  each  project.  Figure  45  shows  the  same  plot 
without  rounding  the  PME  scores.  The  higher  the  project  success  rating  for  a  project,  the 
higher  the  project  management  effectiveness  score  is  for  that  project.  Project 
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management  effectiveness  is  one  of  the  factors  affecting  the  project  success.  There  are 
other  factors  to  explain  the  differences  between  the  scores  and  the  ratings.  Such  factors 
include  choosing  the  right  problem  to  solve  with  the  development  effort,  the  user  and 
customer  satisfaction  with  the  developed  system  after  the  delivery,  other  stakeholders’ 
satisfaction  with  the  result  of  development  effort  (whether  the  return  of  investment  is 
satisfactory  or  not;  whether  it  is  a  pleasant  learning  experience  for  the  development 
team),  the  market  share  gained  with  the  introduction  of  the  product  to  the  market,  etc. 
Identifying  the  right  problem  to  solve  is  very  important.  Every  project  development  effort 
starts  with  a  need  to  solve  a  problem.  If  this  problem  is  not  identified  correctly,  no  matter 
how  effective  the  project  management  is,  the  solution  would  not  be  successful.  There  are 
examples  of  this  situation  especially  in  government  projects.  The  projects  and  developed 
systems  are  shelved  without  being  deployed  because  the  need  is  not  adequately  satisfied, 
or  even  just  for  political  reasons. 

Project  Success  and  Rounded  PME  Scores 


— *—  Project  Success  Score  Rounded  PME  Score 


Figure  44.  Project  Success  and  Rounded  PME  Score  for  each  Project  in  the  Dataset 
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Project  Success  and  PME  Scores 


— Project  Success  Score  — PME  Score 


Figure  45.  Project  Success  and  PME  Score  for  each  Project  in  the  Dataset 

In  the  rest  of  the  section,  various  tables  show  the  project  management  area  scores, 
project  management  effectiveness  scores,  and  other  basic  statistics  regarding  the  dataset. 
These  basic  statistics  include  minimum,  maximum,  range,  mean,  standard  deviation  and 
variation  of  scores. 

Table  41  presents  all  the  scores  computed  from  the  data  gathered  in  the  survey 
studies.  In  Table  42,  the  main  project  management  area  scores,  project  management 
effectiveness  (PME)  scores,  success  ratings  gathered  from  study  participants,  and  basic 
statistical  data  are  presented. 

The  people  area  has  the  highest  minimum  and  maximum  scores  among  all  the 
area  scores.  The  lowest  people  area  score  is  5.7  and  the  highest  people  area  score  is  9.7. 
The  mean  of  people  area  scores  is  also  the  highest  in  the  dataset  at  6.9.  The  standard 
deviation  and  the  variance  for  people  area  scores  are  close  to  1 . 

The  minimum  process  area  score  is  also  high  when  compared  to  other  area  scores. 
The  range  of  process  area  scores  are  close  to  the  range  of  people  area  scores  and  project 
management  effectiveness  scores.  Like  the  people  area  scores,  the  standard  deviation  and 
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the  variance  are  close  to  1 .  It  is  important  to  note  the  similarities  between  the  process  area 
scores  and  project  management  effectiveness  scores.  Almost  all  of  the  statistical  data  is 
identical  for  these  two  sets  of  scores. 

The  set  of  product  scores  in  the  dataset  have  a  higher  variance.  The  range  of 
product  area  scores  are  also  the  highest.  On  the  other  hand,  the  mean  of  product  scores 
are  close  to  the  mean  of  project  management  effectiveness  scores. 

Risk  scores  are  distinctly  lower  than  the  other  area  scores.  The  mean  of  product 
scores  are  5.6.  As  indicated  by  the  study  participants  during  the  interviews,  risk 
management  in  software  projects  is  not  getting  the  necessary  focus  and  attention  it 
deserves.  Therefore,  many  project  development  efforts  have  significant  room  for 
improvement  in  risk  management. 

The  lowest  PME  score  in  this  dataset  is  5.2,  while  the  highest  PME  score  is  8.7. 
The  mean  of  the  project  management  effectiveness  (PME)  scores  in  the  dataset  is  6.6.  It 
is  fair  to  say  that  most  of  the  projects  in  this  dataset  are  successful  projects  with  effective 
project  management.  The  standard  deviation  is  0.96,  which  is  very  close  to  1.  The 
standard  deviation  and  variance  of  PME  scores  are  close  to  the  standard  deviation  and 
variance  of  people  and  process  scores. 

Project  management  effectiveness  (PME)  scores  are  rounded  because  most 
executives  are  interested  in  simple  but  informative  metrics.  The  difference  between  a 
PME  score  of  6.2  and  a  PME  score  of  6  is  not  very  important  for  the  purposes  of 
overseeing  a  project  for  a  high-level  executive.  That  is  why  PME  scores  are  rounded  to 
provide  a  simpler  metric.  The  set  of  scores  obtained  by  rounding  PME  scores  have 
similar  statistics  with  PME  scores.  The  mean  of  rounded  PME  (PME-R)  scores  are  6.8 
while  the  mean  of  PME  scores  is  6.6.  The  standard  deviation  and  variance  of  PME-R 
scores  is  1 . 

The  set  of  project  success  ratings  was  also  analyzed.  The  mean  of  this  set  is  7.6, 
which  is  high.  That  is  why  it  is  concluded  that  most  of  the  projects  in  this  dataset  are 
considered  successful  projects.  It  is  possible  that  most  of  the  study  participants  preferred 
to  report  a  successful  project  for  the  survey  study.  That  may  be  the  reason  for  having 
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such  a  high  mean  for  this  set  of  scores.  There  is  only  one  project  that  has  a  lower  success 
rating  than  5  in  this  dataset.  More  survey  studies  with  low  success  ratings  may  provide 
new  insights  in  future  studies. 

Tables  43  through  46  all  report  the  statistical  data  for  project  management  area 
scores.  They  will  not  be  discussed  in  detail  here,  since  it  is  not  within  the  scope  of  this 
research.  It  is  only  provided  here  for  guiding  future  studies. 
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PROJECT  IDENTIFIER 

A 

B 

C 

D 

E 

F 

G 

H 

I 

J 

K 

L 

M 

N 

O 

P 

PEOPLE  AREA 

Communication 

6.5 

7.2 

6.4 

7.1 

6.0 

6.1 

5.0 

6.2 

7.8 

8.8 

6.8 

5.1 

6.3 

9.2 

6.3 

8.8 

Teamwork 

6.1 

7.8 

6.1 

7.1 

7.1 

6.4 

5.9 

5.7 

8.0 

7.8 

6.9 

5.7 

7.6 

7.6 

6.5 

9.6 

Leadership 

5.9 

8.4 

6.2 

6.6 

5.0 

7.9 

5.9 

5.9 

6.9 

7.9 

7.4 

6.6 

8.1 

9.1 

6.3 

9.1 

Organizational  Commitment 

5.4 

7.3 

7.6 

6.7 

8.1 

6.4 

5.7 

5.3 

7.5 

8.0 

6.9 

6.5 

7.4 

6.6 

7.9 

9.8 

Project  Manager 

7.1 

8.0 

5.2 

8.1 

7.1 

7.7 

6.6 

6.4 

9.1 

8.6 

7.4 

6.3 

7.8 

9.2 

8.1 

9.5 

Stakeholder  Involvement 

7.1 

6.2 

7.7 

7.7 

5.4 

3.4 

5.4 

7.2 

8.9 

6.8 

6.0 

4.2 

6.3 

7.5 

6.5 

8.3 

Staffing  and  Hiring 

4.4 

7.3 

5.9 

6.0 

6.1 

6.3 

6.6 

4.6 

7.2 

6.8 

6.5 

5.7 

6.4 

5.8 

7.9 

9.8 

PEOPLE  SCORE 

6.1 

7.5 

6.4 

7.0 

6.4 

6.3 

5.9 

5.9 

7.9 

7.8 

6.8 

5.7 

7.1 

7.9 

7.1 

9.3 

PROCESS  AREA 

Requirements  Management 

7.0 

7.1 

7.2 

5.4 

3.8 

4.8 

7.0 

5.3 

7.3 

6.8 

7.2 

6.1 

6.1 

8.0 

9.2 

9.7 

Project  Monitoring  and  Control 

7.6 

5.5 

6.2 

6.9 

5.2 

6.2 

7.0 

5.1 

7.9 

7.0 

6.2 

5.6 

6.4 

8.1 

5.8 

8.1 

Project  Planning  and  Estimation 

6.4 

6.8 

5.6 

6.7 

6.7 

4.9 

6.3 

6.2 

7.9 

7.4 

6.6 

5.2 

5.9 

7.2 

7.1 

8.6 

Scope  Management 

6.9 

7.0 

6.1 

5.9 

3.9 

4.9 

6.1 

3.7 

6.9 

5.8 

6.3 

3.4 

5.8 

7.7 

5.6 

7.9 

PROCESS  SCORE 

7.0 

6.6 

6.3 

6.2 

4.9 

5.2 

6.6 

5.1 

7.5 

6.7 

6.6 

5.1 

6.0 

7.8 

6.9 

8.6 

PRODUCT  AREA 

Configuration  Management 

8.7 

7.2 

4.5 

2.2 

2.2 

4.0 

8.2 

5.5 

8.5 

5.5 

8.7 

8.0 

5.0 

7.2 

8.2 

9.3 

Quality  Engineering 

7.1 

8.2 

7.8 

5.6 

7.1 

7.2 

7.5 

5.5 

8.1 

8.4 

6.9 

6.2 

5.4 

7.3 

6.8 

9.7 

PRODUCT  SCORE 

7.9 

7.7 

6.2 

3.9 

4.6 

5.6 

7.8 

5.5 

8.3 

6.9 

7.8 

7.1 

5.2 

7.2 

7.5 

9.5 

RISK  AREA 

Risk  Assessment 

6.4 

5.6 

5.5 

5.5 

3.7 

5.6 

6.8 

5.0 

7.6 

4.9 

6.4 

3.8 

6.2 

8.1 

6.2 

8.5 

Risk  Control 

6.3 

5.5 

4.4 

5.7 

3.7 

5.4 

5.9 

6.3 

8.1 

6.1 

4.6 

3.7 

5.0 

8.0 

5.0 

5.6 

RISK  SCORE 

6.3 

5.5 

5.0 

5.6 

3.7 

5.5 

6.4 

5.7 

7.9 

5.5 

5.5 

3.7 

5.6 

8.0 

5.6 

7.0 

PME  SCORE 

6.7 

6.9 

6.1 

5.9 

5.1 

5.7 

6.6 

5.5 

7.9 

6.9 

6.7 

5.5 

6.1 

7.7 

6.9 

8.7 

ROUNDED  PME  SCORE 

7 

7 

6 

6 

5 

6 

7 

6 

8 

7 

7 

6 

6 

8 

7 

9 

PROJECT  SUCCESS  RATING 

6 

9 

5 

7 

3 

7 

8 

7 

10 

10 

9 

7 

8 

9 

7 

9 

Table  4 1 .  Project  Management  Area  Scores 
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PEOPLE 

PROCESS 

PRODUCT 

RISK 

PME 

PME-R 

Success 

Rating 

Project  A 

6.1 

7.0 

7.9 

6.3 

6.7 

7 

6 

Project  B 

7.5 

6.6 

7.0 

5.8 

6.8 

7 

9 

Project  C 

6.4 

6.3 

6.2 

5.0 

6.1 

6 

5 

Project  D 

7.0 

6.2 

3.9 

5.6 

5.9 

6 

7 

Project  E 

6.4 

5.1 

4.6 

3.7 

5.2 

5 

3 

Project  F 

6.3 

5.3 

5.6 

5.5 

5.7 

6 

7 

Project  G 

5.9 

6.6 

7.8 

6.4 

6.6 

7 

8 

Project  H 

5.9 

5.1 

5.5 

5.7 

5.5 

6 

7 

Project  I 

7.9 

7.5 

8.3 

7.9 

7.9 

8 

10 

Project  J 

7.8 

6.7 

6.9 

5.5 

6.9 

7 

10 

Project  K 

6.8 

6.6 

7.8 

5.5 

6.7 

7 

9 

Project  L 

5.7 

5.1 

7.1 

3.7 

5.5 

6 

7 

Project  M 

7.1 

6.0 

5.2 

5.6 

6.1 

6 

8 

Project  N 

7.9 

7.8 

7.2 

8.0 

7.8 

8 

9 

Project  O 

7.1 

6.9 

7.5 

5.6 

6.9 

7 

7 

Project  P 

9.3 

8.6 

9.5 

7.0 

8.7 

9 

9 

Min 

5.7 

5.1 

3.9 

3.7 

5.2 

5.0 

3.0 

Max 

9.3 

8.6 

9.5 

8.0 

8.7 

9.0 

10.0 

Range 

3.6 

3.5 

5.6 

4.3 

3.5 

4.0 

7.0 

Mean 

6.9 

6.5 

6.8 

5.8 

6.6 

6.8 

7.6 

Standard  Deviation 

0.96 

1.01 

1.48 

1.19 

0.96 

1.00 

1.86 

Variance 

0.92 

1.02 

2.19 

1.41 

0.92 

1.00 

3.46 

Table  42.  Data  Analysis  of  Main  Area  Scores 
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C 

T 

Project  A 

6.5 

6.1 

Project  B 

7.2 

7.8 

Project  C 

6.4 

6.1 

Project  D 

7.1 

7.1 

Project  E 

6.3 

7.1 

Project  F 

6.1 

6.4 

Project  G 

5.0 

5.9 

Project  H 

6.2 

5.7 

Project  I 

7.8 

8.0 

Project  J 

8.8 

7.8 

Project  K 

6.8 

6.9 

Project  L 

5.1 

5.7 

Project  M 

6.3 

7.6 

Project  N 

9.2 

7.6 

Project  O 

6.3 

6.5 

Project  P 

8.8 

9.6 

Min 

5.0 

5.7 

Max 

9.2 

9.6 

Range 

4.2 

3.9 

Mean 

6.9 

7.0 

Standard  Deviation 

1.24 

1.05 

Variation 

1.55 

1.11 

Table  43. 


Requirements 

Management 

Project 

Monitoring  and 
Control 

Project  Planning 
and  Estimation 

Scope 

Management 

Process 

Project  A 

7.0 

7.6 

6.4 

6.9 

7.0 

Project  B 

7.1 

5.5 

6.8 

7.0 

6.6 

Project  C 

7.2 

6.2 

5.6 

6.1 

6.3 

Project  D 

5.4 

6.9 

6.7 

5.9 

6.2 

Project  E 

3.8 

5.2 

7.1 

4.2 

5.1 

Project  F 

4.8 

6.6 

4.9 

4.9 

5.3 

Project  G 

7.0 

7.0 

6.3 

6.1 

6.6 

Project  H 

5.3 

5.1 

6.2 

3.7 

5.1 

Project  I 

7.3 

7.9 

7.9 

6.9 

7.5 

Project  J 

6.8 

7.0 

7.4 

5.8 

6.7 

Project  K 

7.2 

6.2 

6.6 

6.3 

6.6 

Project  L 

6.1 

5.6 

5.2 

3.4 

5.1 

Project  M 

6.1 

6.4 

5.9 

5.8 

6.0 

Project  N 

8.0 

8.1 

7.3 

7.7 

7.8 

Project  O 

9.2 

5.8 

7.1 

5.6 

6.9 

Project  P 

9.7 

8.1 

8.6 

7.9 

8.6 

Min 

3.8 

5.1 

4.9 

3.4 

5.1 

Max 

9.7 

8.1 

8.6 

7.9 

8.6 

Range 

5.9 

3.0 

3.7 

4.5 

3.5 

Mean 

6.7 

6.6 

6.6 

5.9 

6.5 

Standard  Deviation 

1.52 

1.00 

0.96 

1.32 

1.01 

Variation 

2.30 

1.01 

0.92 

1.74 

1.02 

Table  44.  Data  Analysis  of  Process  Area  Scores 
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Configuration 

Management 

Quality 

Engineering 

Product 

Project  A 

8.7 

7.1 

7.9 

Project  B 

7.2 

6.9 

7.0 

Project  C 

4.5 

7.8 

6.2 

Project  D 

2.2 

5.6 

3.9 

Project  E 

2.2 

7.1 

4.6 

Project  F 

4.0 

7.2 

5.6 

Project  G 

8.2 

7.5 

7.8 

Project  H 

5.5 

5.5 

5.5 

Project  I 

8.5 

8.1 

8.3 

Project  J 

5.5 

8.4 

6.9 

Project  K 

8.7 

6.9 

7.8 

Project  L 

8.0 

6.2 

7.1 

Project  M 

5.0 

5.4 

5.2 

Project  N 

7.2 

7.3 

7.2 

Project  O 

8.2 

6.8 

7.5 

Project  P 

9.3 

9.7 

9.5 

Min 

2.2 

5.4 

3.9 

Max 

9.3 

9.7 

9.5 

Range 

7.2 

4.3 

5.6 

Mean 

6.4 

7.1 

6.8 

Standard  Deviation 

2.34 

1.12 

1.48 

Variation 

5.47 

1.26 

2.19 

Table  45.  Data  Analysis  of  Product  Area  Scores 
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Risk  Assessment 

Risk  Control 

Risk 

Project  A 

6.4 

6.3 

6.3 

Project  B 

5.6 

5.9 

5.8 

Project  C 

5.5 

4.4 

5.0 

Project  D 

5.5 

5.7 

5.6 

Project  E 

3.7 

3.7 

3.7 

Project  F 

5.6 

5.4 

5.5 

Project  G 

6.8 

5.9 

6.4 

Project  H 

5.0 

6.3 

5.7 

Project  I 

7.6 

8.1 

7.9 

Project  J 

4.9 

6.1 

5.5 

Project  K 

6.4 

4.6 

5.5 

Project  L 

3.8 

3.7 

3.7 

Project  M 

6.2 

5.0 

5.6 

Project  N 

8.1 

8.0 

8.0 

Project  O 

6.2 

5.0 

5.6 

Project  P 

8.5 

5.6 

7.0 

Min 

3.7 

3.7 

3.7 

Max 

8.5 

8.1 

8.0 

Range 

4.7 

4.4 

4.3 

Mean 

6.0 

5.6 

5.8 

Standard  Deviation 

1.35 

1.26 

1.19 

Variation 

1.82 

1.59 

1.41 

Table  46.  Data  Analysis  of  Risk  Area  Scores 


In  order  to  understand  the  measure  of  association  between  project  success  ratings 
provided  by  the  study  participants  and  the  project  management  effectiveness  (PME) 
metric  for  the  project,  correlation  analysis  was  conducted.  The  choice  of  analysis  method 
was  parametric  bivariate  correlation  analysis.  This  choice  was  driven  by  the  applicability 
and  suitability  of  the  assumptions  rather  than  being  a  preference.  This  method  was 
suitable  for  statistical  association  analysis  in  this  particular  study.  The  parametric 
correlation  measure  most  often  used  is  the  Pearson  product  moment  coefficient,  r 
(Emory,  1980).  This  coefficient  of  correlation  is  the  summary  statistic  that  represents  the 
linear  relationship  between  two  sets  of  variables  of  interest  (Emory,  1980).  In  this 
research,  that  is  exactly  what  we  are  looking  for  and  this  is  what  is  needed  to  test  the 
research  hypothesis.  Overall,  the  analysis  showed  that  there  is  a  strong  positive 
correlation  r  ,  between  the  software  project  management  effectiveness  metric  proposed  in 
this  research  and  the  software  project  success  rating  provided  by  the  study  participants. 
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In  statistics,  the  Pearson  product-moment  correlation  coefficient  is  a  commonly 
used  measure  to  identify  the  linear  relationship  between  the  sets  of  variables.  It  is 
sometimes  referred  as  MCV  or  PMCC.  According  to  the  usual  convention,  when  the 
Pearson  product-moment  correlation  coefficient  is  calculated  for  the  entire  population, 
the  Greek  letter  rho  (p)  is  used.  When  the  correlation  coefficient  is  calculated  for  a 
sample,  then  Latin  letter  r  is  used.  The  Pearson  correlation  coefficient,  r,  ranges  from  -1 
to  +1.  The  sign  of  the  correlation  coefficient  indicates  the  direction  of  the  linear  relation. 
Correlation  coefficient  +1  indicates  that  there  is  a  perfect  positive  correlation  between 
two  variables.  If  the  correlation  coefficient  is  -1,  then  there  is  a  perfect  negative  relation 
between  the  variables  of  interest.  Perfect  relationships  are  rarefy  observed  in  social 
studies.  In  a  positive  correlation,  when  one  variable  goes  up  the  other  variable  goes  up  as 
well.  In  a  negative  correlation,  when  one  variable  goes  up,  the  other  variable  goes  down. 
There  is  also  one  special  case,  and  that  is  when  the  correlation  coefficient  is  zero.  In  that 
case,  there  is  no  linear  relationship  between  the  variables.  The  absolute  value  of  the 
correlation  coefficient  indicates  the  strength  of  the  relationship.  The  higher  the  value  of 
the  correlation  coefficient,  the  stronger  the  relationship  between  the  variables  are.  What 
constitutes  as  a  strong  correlation  depends  strictly  on  the  research  being  conducted.  In 
social  studies,  as  a  rule  of  thumb,  when  the  Pearson  product-moment  correlation 
coefficient  is  higher  than  0.5,  then  it  may  be  assumed  that  there  is  strong  correlation 
between  the  variables. 

In  statistics,  r  ,  the  square  of  coefficient  correlation  r,  is  called  the  coefficient  of 
determination.  The  coefficient  of  determination,  r  ,  is  the  percentage  of  variance  in  one 
variable  explained  by  the  linear  relationship  with  the  other  variable.  This  r  basically 
refers  to  the  amount  of  variation  in  one  variable  explained  by  the  relationship  between 
variables. 

Table  47  shows  the  correlation,  r,  between  main  area  scores,  project  management 
effectiveness  (PME)  scores,  and  project  success  ratings.  All  scores  have  strong  positive 
correlations  between  each  other.  Only  the  correlation  between  the  people  and  product 
main  area  scores  is  0.42.  While  the  correlation  between  these  two  sets  of  scores  may  not 
be  strong,  there  is  certainly  a  relationship  of  interest  between  the  people  and  product 
main  area  scores.  In  this  research,  our  main  interest  is  the  relationship  between  project 
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success  and  the  project  management  effectiveness  metric  proposed  with  this  research.  It 
is  observed  that  there  is  a  strong  positive  correlation  between  project  success  and  project 
management  effectiveness.  The  Pearson  product-moment  correlation  coefficient,  r, 
between  project  success  rating  and  rounded  project  management  effectiveness  (PME-R) 
is  0.73.  The  coefficient  of  determination,  r  ,  for  these  two  sets  of  metrics  is  0.53.  In 
simple  terms,  this  means  that  project  management  effectiveness  accounts  for  half  the 
variation  in  project  success.  This  result  supports  the  hypothesis  stated  in  the  beginning  of 
this  research. 

It  is  also  very  important  to  note  that  there  is  no  weak  or  strong  negative 
correlation  between  the  set  of  scores.  Negative  correlations  between  these  scores  were 
not  expected  and  there  were  not  any.  Such  a  result  supports  the  claim  that  this  research  is 
sound,  especially  from  the  point  of  construct  validity. 

All  main  area  scores  have  strong  correlation  with  the  project  success.  Therefore,  it 
is  possible  to  suggest  that  effectiveness  in  each  main  area  has  an  effect  on  project 
success. 

The  main  people  area  scores  have  strong  correlation  with  the  main  process  area 
scores  and  project  management  effectiveness  (PME)  scores.  The  Pearson’s  correlation 
coefficient  between  people  area  scores  and  process  area  scores  is  0.80.  The  correlation,  r, 
between  people  area  scores  and  PME  scores  is  0.83.  The  strength  of  these  correlations  is 
noteworthy. 

One  notable  aspect  of  the  process  area  scores  is  that  they  have  very  strong 
correlations  with  all  other  main  areas.  The  correlation,  r,  between  the  people  and  process 
area  scores  is  0.80,  while  r  between  the  process  and  product  area  scores  is  0.72,  and  r 
between  process  and  risk  main  area  scores  is  0.80.  The  correlation  between  process  area 
scores  and  project  management  effectiveness  (PME)  scores  is  almost  perfect  at  0.97.  This 
is  extraordinary.  This  means  that  it  is  possible  to  predict  the  PME  metric  using  process 
area  scores.  However,  it  does  not  mean  that  effectiveness  in  only  the  process  area  leads  to 
effectiveness  in  project  management  because  the  Pearson  correlation  coefficient  does  not 
necessarily  indicate  causality.  There  may  be  other  factors  affecting  the  correlation.  In  this 
case,  people,  product  and  risk  management  effectiveness  contribute  the  PME  metric. 
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Positive  strong  correlations  between  all  these  metrics  are  indications  of  the 
soundness  of  this  research.  Table  48  presents  the  correlations  between  project 
management  area  scores,  main  area  scores,  PME  scores,  and  project  success  ratings.  All 
project  management  area  scores  have  positive  strong  correlations  with  the  project 
management  effectiveness  metric.  The  strong  positive  correlations  between  project 
management  area  scores  and  people  main  area  scores  are  noteworthy.  There  is  only  one 
exception  and  that  is  configuration  management  scores. 

All  project  management  area  scores  except  organizational  commitment, 
stakeholder  involvement,  project  planning  and  estimation,  and  quality  engineering  area 
scores  have  strong  correlation  with  project  success  ratings.  The  exception  in  these  areas 
does  not  necessarily  mean  that  there  is  no  relationship  with  project  success  because  the 
Pearson  product-moment  correlation  coefficient  is  about  the  linear  relationship  between 
two  sets  of  variables.  The  relations  between  these  metrics  may  be  non-linear.  Further 
research  is  required  to  identify  the  type  of  relationships  between  these  sets  of  metrics. 
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PEOPLE 

PROCESS 

PRODUCT 

RISK 

PME 

PME-R 

PROJECT 

SUCCESS 

RATING 

PEOPLE 

* 

0.80 

0.42 

0.57 

0.83 

0.75 

0.59 

PROCESS 

* 

0.72 

0.80 

0.97 

0.93 

0.58 

PRODUCT 

* 

0.53 

0.79 

0.85 

0.54 

RISK 

* 

0.82 

0.82 

0.64 

PME 

* 

0.98 

0.68 

PME-R 

* 

0.73 

Table  47.  Correlation  Coefficient  of  Main  Project  Management  Areas  (Project  A  to  Project  P) 
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PEOPLE 

PROCESS 

PRODUCT 

RISK 

PME 

PME-R 

PROJECT 

SUCCESS 

RATING 

Communication 

0.87 

0.72 

0.30 

0.61 

0.74 

0.66 

0.53 

Teamwork 

0.96 

0.68 

0.33 

0.47 

0.73 

0.63 

0.52 

Leadership 

0.73 

0.55 

0.33 

0.49 

0.62 

0.61 

0.71 

Organizational  Commitment 

0.76 

0.44 

0.23 

0.00 

0.46 

0.34 

0.13 

Project  Manager 

0.85 

0.71 

0.36 

0.68 

0.76 

0.72 

0.69 

Stakeholder  Involvement 

0.60 

0.66 

0.25 

0.60 

0.61 

0.54 

0.28 

Staffing  and  Hiring 

0.76 

0.59 

0.50 

0.25 

0.64 

0.59 

0.42 

Requirements  Management 

0.61 

0.85 

0.81 

0.58 

0.84 

0.84 

0.52 

Project  Monitoring  and  Control 

0.58 

0.80 

0.51 

0.81 

0.77 

0.76 

0.50 

Project  Planning  and  Estimation 

0.81 

0.76 

0.46 

0.55 

0.77 

0.70 

0.40 

Scope  Management 

0.71 

0.92 

0.56 

0.78 

0.86 

0.80 

0.52 

Configuration  Management 

0.25 

0.60 

0.94 

0.48 

0.67 

0.76 

0.55 

Quality  Engineering 

0.59 

0.66 

0.69 

0.38 

0.70 

0.66 

0.28 

Risk  Assessment 

0.63 

0.88 

0.63 

0.92 

0.87 

0.87 

0.56 

Risk  Control 

0.41 

0.57 

0.31 

0.90 

0.61 

0.62 

0.61 

Table  48.  Correlation  Coefficient  of  Main  Areas  to  Project  Management  Areas  (Project  A  to  Project  P) 
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Table  50  presents  the  Pearson  product  moment  correlation  coefficients,  r,  between 
project  management  areas  in  the  dataset.  Any  r  values  higher  than  0.5  are  indicated  in 
bold  letters.  It  was  identified  that  many  project  management  area  scores  are  strongly 
correlated  with  each  other.  Such  results  show  the  soundness  of  this  research  study.  For 
example,  the  correlation  between  communication  and  teamwork  is  0.79.  This  is  a  strong 
positive  correlation.  Such  a  relation  between  communication  and  teamwork  has  already 
been  established  in  studies  conducted  prior.  The  particular  relationships  between  project 
management  areas  are  not  within  the  scope  of  this  doctoral  study.  The  relationships 
between  two  areas  or  a  set  of  areas  are  potential  doctoral  research  topics.  The  results  in 
this  table  provide  important  guidance  for  future  research.  In  addition,  more  studies  on 
software  projects  using  the  software  project  management  evaluation  instrument  (SPMEI) 
will  help  to  investigate  the  relationships  with  more  accuracy  and  precision. 

Table  51  presents  the  correlations  between  project  management  areas  based  on 
the  data  from  all  survey  studies.  The  projects  data  excluded  from  the  dataset  may  be  used 
in  this  table.  The  project  management  effectiveness  (PME)  metric  may  not  be  applicable 
to  the  excluded  projects,  but  the  relationships  between  project  management  area  scores 
should  still  be  applicable.  It  is  observed  that  the  strength  of  relationships  r  values  is 
higher  when  data  from  all  projects  are  included. 


Project  Management  Area 
(PMA) 

Identifier 

Project  Management  Area 
(PMA) 

Identifier 

Communication 

C 

Project  Monitoring  and  Control 

PMC 

Teamwork 

T 

Project  Planning  and  Estimation 

PPE 

Leadership 

L 

Scope  Management 

SM 

Organizational  Commitment 

OC 

Configuration  Management 

CM 

Project  Manager 

PM 

Quality  Engineering 

QE 

Stakeholder  Involvement 

SI 

Risk  Assessment 

RA 

Staffing  and  Hiring 

S 

Risk  Control 

RC 

Requirements  Management 

RM 

Table  49.  Identifiers  for  Project  Management  Areas 
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Table  50.  Correlation  Coefficient  of  Project  Management  Areas  (Project  A  to  Project  O) 


OC  PM 


CM 

QE 

RA 

RC 

0.42 

0.72 

0.71 

0.62 

0.39 

0.67 

0.65 

0.47 

0.28 

0.33 

0.41 

0.25 

0.24 

0.62 

0.41 

0.08 

0.39 

0.47 

0.57 

0.58 

0.50 

0.68 

0.77 

0.71 

0.51 

0.68 

0.60 

0.28 

0.82 

0.73 

0.82 

0.53 

0.67 

0.88 

0.91 

0.81 

0.64 

0.87 

0.81 

0.74 

0.66 

0.76 

0.89 

0.76 

* 

0.67 

0.75 

0.57 

* 

0.79 

0.63 

* 

0.79 

Table  5 1 .  Correlation  Coefficient  of  Project  Management  Areas  (All  20  Projects) 


E.  SOUNDNESS  OF  THE  MEASUREMENT  STUDY 

In  this  section,  the  soundness  of  the  measurement  study  will  be  discussed.  First, 
the  sources  of  errors  and  their  implications  on  the  study  will  be  outlined.  Then,  validity, 
reliability  and  practicality  of  the  measurement  study  will  be  discussed  briefly. 

According  to  Emory  (1980),  there  are  three  major  considerations  for  evaluating  a 
measurement  tool.  They  are  validity,  reliability  and  practicality.  These  terms  may  be 
described  as  follows: 

Validity  refers  to  the  extent  to  which  a  test  measures  what  we  actually 
wish  to  measure.  Reliability  has  to  do  with  the  accuracy  and  precision  of  a 
measurement  procedure....  Practicality  is  concerned  with  a  wide  range  of 
factors  of  economy,  convenience,  and  interpretability  (Emory,  1980). 

1.  Sources  of  Measurement  Differences 

It  is  very  hard  to  develop  an  ideal  measurement  tool  without  contamination  from 
the  sources  of  errors.  Therefore,  it  is  important  to  recognize  these  sources  of  errors. 
Whenever  possible,  these  sources  of  errors  should  be  identified,  eliminated  or 
neutralized.  If  elimination  is  not  feasible  or  sometimes  possible,  then  the  sources  of  errors 
should  at  least  be  acknowledged  so  that  the  users  of  the  measurement  tool  know  how 
accurate  and  precise  the  measurement  activity  is. 

According  to  Emory  (1980),  there  are  four  major  error  sources  that  may 
contaminate  the  results.  These  sources  are  the  respondent,  the  situation,  the  measurer  and 
the  instrument. 


a.  The  Respondent  as  an  Error  Source 

The  study  participants  may  be  a  source  of  errors  for  many  reasons.  These 
reasons  include  personal  bias,  social  class,  ethnic  background,  etc.  In  addition,  the 
respondent  may  be  affected  by  various  conditions  such  as  fatigue,  boredom,  anxiety,  etc. 
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Every  research  is  unique  in  various  aspects.  Common  causes  of  errors 
introduced  by  participants  observed  in  many  other  studies  do  not  play  a  role  in  this 
research  because  of  its  nature.  For  example  ethnic  background,  social  class,  etc. 

One  of  most  obvious  characteristics  of  the  study  participants  in  this 
research  is  that  they  are  highly  educated.  Most  of  the  participants  are  quite  experienced  in 
the  software  field  as  well.  Some  of  the  study  participants  had  more  than  twenty  years  of 
software  and  systems  development  experience.  All  of  the  study  participants  had  at  least  a 
certain  amount  of  management  experience  in  their  careers. 

In  this  research,  the  most  likely  source  of  errors  by  the  study  respondents 
may  be  due  to  the  bias  resulting  from  personal  views  of  software  project  management. 
However,  the  participation  in  the  research  was  voluntary.  The  types  of  practitioners  who 
volunteer  in  these  research  studies  tend  to  be  objective.  Generally,  they  want  to  improve 
themselves,  improve  their  work  practices,  and  try  to  leam  something  new.  So,  they  also 
try  to  be  as  objective  as  possible  to  get  the  most  out  of  their  participation. 

The  proposed  metric  with  this  research  is  designed  to  assess  the  project 
management  effectiveness  not  the  effectiveness  of  various  people  in  the  project 
organizations.  The  study  participants  are  ensured  of  this  particular  point.  This  is  to  avoid 
any  misunderstandings  and  contamination  of  results  due  to  the  bias  that  might  occur 
because  the  participants  think  they  are  being  evaluated.  In  order  to  eliminate  or  at  least 
reduce  any  possible  bias,  the  questions  in  the  software  project  management  evaluation 
instrument  (SPMEI)  are  carefully  crafted  so  that  the  wording  clearly  reflects  what  is 
being  evaluated. 

During  the  study,  it  was  observed  that  the  participants  did  not  contribute 
as  a  significant  source  of  error.  SPMEI  is  designed  in  such  a  way  that  errors  caused  by 
participants  can  be  identified.  Many  of  the  software  project  management  evaluation 
instrument  (SPMEI)  questions  are  related.  Discrepancies  among  the  responses  may  be 
easily  identified. 

In  this  study,  the  study  participants  provided  the  project  success  ratings 
based  on  a  scale  from  0  to  10.  With  the  current  state-of-the-art  in  software  project 
management,  this  is  still  one  of  the  best  ways  to  detennine  the  success  of  a  project.  This 
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method  has  been  used  in  many  other  studies  as  well.  As  indicated  earlier  in  previous 
sections,  there  is  currently  no  established  method  to  determine  the  success  of  a  software 
project.  At  the  beginning  of  the  study,  one  of  the  concerns  posed  was  the  possibility  of 
study  participants  rating  their  projects  with  high  ratings.  The  same  issue  was  a  concern  in 
a  similar  study  conducted  previously  by  Osmundson  et  al.  (2003).  In  that  study,  the  case 
that  the  study  participants  were  rating  the  project  success  consistently  higher  was  not 
observed.  In  this  study,  it  was  observed  that  study  participants  tried  to  rate  the  projects  as 
objectively  as  possible. 

b.  The  Measurer  as  an  Error  Source 

Some  research  designs  require  the  researcher  or  the  measurer  to  be  present 
during  the  study.  In  some  cases,  the  study  participants  may  get  anxious  because  the 
measurer  is  present.  They  may  try  to  impress  the  measurer  or  the  researchers,  or  they  may 
get  nervous  and  behave  differently  than  nonnal.  Therefore,  the  researchers  should  be 
aware  of  these  types  of  error  sources  in  research  designs  requiring  the  researcher  to  be 
present  during  the  study.  In  order  to  eliminate  this  type  of  contamination,  the  author  was 
not  with  the  study  participants  during  the  process.  There  was  one  exception  to  this:  in  one 
of  the  survey  studies  at  the  beginning  of  the  research,  the  author  was  present  when  the 
study  participants  completed  the  SPMEI.  This  was  done  deliberately.  The  goal  was  to 
observe  the  study  participant  and  identify  any  difficulties  in  the  surveying  protocol.  It 
was  observed  that  the  study  participant  was  able  to  complete  the  instrument  without  any 
difficulty.  This  was  important  to  help  understand  the  nature  of  the  study  protocol.  It  is 
important  to  note  that  this  study  participant  was  chosen  carefully.  The  author  and  the 
study  participant  knew  each  other.  This  particular  study  participant  was  comfortable  with 
the  author.  Therefore,  any  contamination  due  to  the  existence  of  researcher  during  the 
procedure  was  minimized  in  this  particular  survey  study. 

The  SPMEI  is  designed  as  a  self-evaluation  tool.  This  was  an  important 
and  deliberate  decision  made  early  in  the  research.  There  are  two  reasons  for  this.  The 
first  reason  is  that  the  bias  due  to  the  measurer  as  a  source  of  error  is  eliminated.  The 
second  reason  relates  to  practicality  issues.  In  this  research,  a  new  project  management 
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measurement  tool  is  introduced.  This  tool  is  designed  to  be  used  by  practitioners.  Thus, 
the  tool  is  designed  in  such  a  way  that  the  software  managers  are  able  to  use  it  by  their 
selves. 

In  this  research,  the  study  participants  completed  the  SPMEI  by 
themselves  on  their  own  time.  The  researcher  was  not  present  during  the  process.  Sources 
of  errors  caused  by  the  measurer  or  researcher  being  present  during  the  studies  are  almost 
nonexistent. 


c.  Situational  Factors 

Any  condition  that  may  put  a  strain  on  the  respondent  may  contaminate 
the  study  results.  For  instance,  the  room  where  the  study  participant  is  located  during  a 
research  study  may  affect  the  study  results.  In  some  studies,  the  participants  are  brought 
to  controlled  environments  for  close  observation.  In  these  cases,  if  the  study  participant 
becomes  uncomfortable  with  the  environment,  the  results  may  be  contaminated  unless  the 
researchers  are  not  especially  investigating  that  particular  effect.  There  are  many  variants 
of  situational  factors  and  they  are  specific  to  the  research  design.  The  researchers  have  to 
be  aware  of  such  situational  factors.  If  unwanted,  the  researchers  should  eliminate  or  at 
least  account  for  these  factors.  In  this  research,  no  situational  factors  affecting  the 
research  results  are  observed. 

d.  The  Instrument  as  an  Error  Source 

It  is  very  hard  to  design  an  ideal  instrument  in  social  studies.  In  most 
cases,  social  concepts  are  hard  to  capture  and  measure  with  an  instrument.  This  research 
deals  with  the  project  management  aspect  of  software  projects.  There  are  many  social 
issues  involved  in  managing  a  software  project.  Therefore,  development  of  an  instrument 
for  measuring  project  management  effectiveness  was  one  of  the  most  challenging  aspects 
of  this  research.  The  software  project  management  effectiveness  metric  uses  the  software 
project  management  evaluation  instrument  (SPMEI)  and  software  project  management 
evaluation  model  (SPMEM)  to  provide  an  overall  effectiveness  metric.  A  great  deal  of 
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the  effort  in  this  research  went  to  the  development  of  this  instrument.  Extra  effort  was 
spent  in  order  to  minimize  the  errors  in  the  design  of  the  instrument  and  the  evaluation 
model. 

The  SPMEI  questions  were  carefully  designed.  They  are  worded  as  simple 
as  possible.  Most  questions  and  statements  are  concise  and  clear.  The  terms  used  in  the 
questions  and  statements  were  specially  selected  to  minimize  misunderstandings.  The 
software  project  management  evaluation  instrument  was  tested  for  these  issues  during 
pilot  studies.  Study  participants  indicated  that  the  questions  in  the  instrument  were  clear 
and  easily  understandable.  The  analysis  conducted  after  the  survey  studies  showed  that 
there  were  no  significant  errors  caused  by  the  instrument.  On  the  other  hand,  the 
instrument  surely  has  room  for  improvement.  However,  more  research  and  more  samples 
are  required  in  order  to  identify  the  opportunities  for  improvement. 

2.  Validity  of  the  Measurement 

There  are  many  concepts  of  validity  in  research  literature.  There  are  also  many 
categorizations.  Not  all  validity  concepts  or  concerns  are  applicable  to  all  research 
experiments.  Most  validity  related  concerns  were  already  reported  throughout  the 
dissertation  where  they  were  pertinent.  Here,  external  and  internal  validity  issues  related 
to  this  measurement  activity  will  be  discussed  briefly. 

Emory  (1980)  described  external  validity  by  stating  that,  “external  validity  of 
research  findings  refers  to  their  generalizability  across  persons,  settings  and  times.” 

The  software  project  management  effectiveness  (PME)  metric  has  limitations. 
Even  though  the  PME  metric  is  applicable  to  most  software  projects,  it  is  not  applicable 
in  some  software  projects  as  observed  in  the  survey  studies. 

The  software  PME  metric  is  applicable  to  software  or  software  intensive  projects. 
The  metric  is  not  tested  in  other  types  of  projects.  Adaptation  of  the  PME  to  other  types 
of  projects  may  likely  require  modifications  in  the  instrument  and  model. 

The  software  PME  metric  is  independent  from  the  life  cycle  development  model 
used  in  the  projects.  It  is  designed  to  be  applicable  with  almost  all  life  cycle  development 
models.  The  projects  in  the  survey  studies  employed  different  life  cycle  development 
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models  including  agile  development,  waterfall  development,  incremental  development, 
rapid  prototyping  development  model,  and  variants  of  various  other  development  models. 
No  project  employs  a  life  cycle  development  model  as  stated  in  a  textbook.  They  are 
customized  based  on  many  factors.  This  is  one  of  the  reasons  to  design  the  metric 
independent  from  life  cycle  development  models. 

The  PME  metric  is  designed  to  be  used  in  many  types  of  organizations.  Survey 
studies  showed  that  the  metric  is  applicable  to  projects  developed  in  government 
organizations  as  well  as  commercial  organizations.  There  are  cases  in  the  study  in  which 
a  combination  of  various  types  of  organizations  developed  projects.  These  are  among  the 
basic  types  of  organizations.  It  is  observed  that  the  PME  metric  is  applicable  to  these 
environments.  There  are  also  other  types  of  groups  or  organizations  worth  mentioning. 
One  of  them  is  open-source  communities.  The  dataset  does  not  include  an  open-source 
project.  The  author  suspects  that  modifications  may  be  required  in  the  instrument  and 
model  to  ensure  the  applicability  of  the  metric  to  open-source  projects.  Therefore, 
currently  the  applicability  of  the  metric  to  open-source  projects  is  not  known. 

The  PME  metric  is  designed  to  be  applicable  to  all  types  of  product 
developments.  The  products  developed  with  the  projects  in  the  dataset  include  mission 
critical  defense  systems,  embedded  systems,  real  time  systems,  office  automation 
systems,  information  management  systems,  networking  applications,  prototypes,  database 
applications,  etc.  The  survey  studies  covered  many  of  the  product  types. 

The  PME  metric  is  applicable  to  almost  all  project  team  sizes.  The  only  exception 
is  very  small  teams  consisting  of  2-4  people.  The  survey  studies  showed  that  when  the 
project  team  is  very  small  and  the  project  development  is  conducted  in  a  highly  informal 
manner,  the  metric  is  not  applicable.  There  are  three  such  examples.  The  average  number 
of  people  involved  in  these  developments  is  2  to  4.  None  of  these  projects  had  a  project 
plan.  It  is  likely  that  formal  or  semi-formal  project  management  would  not  contribute  to 
these  efforts  significantly.  On  the  other  hand,  it  is  observed  that  when  there  is  a  project 
plan  and  the  development  effort  is  conducted  in  a  formal  or  semi-formal  manner,  the 
metric  is  applicable  even  to  small  teams.  In  two  projects,  the  average  number  of  the 
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people  involved  in  the  project  was  four.  The  metric  is  found  to  be  applicable  to  these 
projects.  In  one  project,  the  project  involved  a  team  of  300  people.  This  is  a  very  large 
project  in  terms  of  number  of  people  involved. 

The  survey  studies  included  projects  developed  in  both  North  America  and 
Europe.  Projects  developed  in  other  parts  of  the  world  were  not  included  in  this  study. 
More  samples,  especially  projects  developed  in  other  geographical  locations  will  be 
beneficial.  Nevertheless,  different  results  are  not  expected. 

All  the  study  participants  in  this  study  are  managers  at  some  level  in  the  project 
organization  or  experts  with  a  broad  view  of  the  project.  The  SPMEI  requires  the 
respondent  to  be  a  project  team  member  who  has  a  broad  view  of  the  project.  The  project 
members  who  assume  a  management  role  are  likely  candidates  to  participate  in  this 
measurement  activity.  The  software  project  management  evaluation  instrument  can  not 
be  completed  by  a  team  member  or  a  stakeholder  with  access  to  only  certain  parts  of  the 
development  effort. 

The  survey  studies  included  examples  from  every  decade  since  1970s.  As 
discussed  earlier,  the  metric  is  not  applicable  to  projects  developed  at  all  timeframes.  It  is 
mostly  applicable  to  projects  developed  after  the  1970s.  The  reasons  are  explained  earlier 
in  detail.  There  may  be  exceptions  to  this  issue.  It  is  observed  that  the  metric  is  applicable 
to  a  project  developed  in  1976.  It  was  applicable  to  this  case  because  this  project  was 
developed  by  a  major  software  company  employing  the  best  management  practices  at  that 
time.  Such  effective  project  management  is  unlikely  in  other  projects  developed  by  many 
of  the  software  development  organizations  during  the  1970s. 

Emory  (1980)  described  internal  validity  as,  “The  ability  of  a  research  instrument 
to  measure  what  it  is  purported  to  measure.”  Establishing  the  internal  validity  of  a  study 
is  not  an  easy  task  because  the  ideal  way  of  understanding  the  internal  validity  is  to 
compare  the  results  of  a  measurement  instrument  with  the  absolute  measures.  In  most 
cases,  the  real  measure  of  what  is  being  measured  is  not  known.  If  it  were  known,  there 
would  not  be  a  need  to  measure  it  in  the  first  place.  So,  the  way  to  overcome  this 
challenge  is  to  seek  other  relevant  evidence  confirming  the  answers  found  with 
measurement  device  (Emory,  1980).  The  key  tenn  here  is  relevant  evidence.  What  is 
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relevant  or  not  depends  on  the  nature  of  the  study  as  well  as  the  judgment  of  the 
researcher.  There  are  three  widely  accepted  classifications  of  internal  validity.  They  are 
content,  criterion-related  and  construct. 

a.  Content  Validity 

The  content  validity  pertains  to  how  well  the  measurement  instrument 
covers  the  concepts  of  interest.  Project  management  is  a  very  broad  concept.  It  is  hard  to 
achieve  full  coverage  with  an  instrument.  However,  in  this  research  a  high  coverage  is 
achieved.  The  software  project  management  evaluation  instrument  is  designed  with  the 
guidance  of  the  software  project  management  framework  proposed  in  this  study.  This 
framework  is  developed  via  an  extensive  literature  search.  Then,  it  is  validated  with  a 
worldwide  survey  of  software  practitioners.  In  the  survey,  the  survey  participants  were 
specifically  asked  about  the  coverage  and  sufficiency  of  the  framework.  A  high 
percentage  of  the  survey  participants  found  the  framework  sufficient.  Furthermore,  the 
SPMEI  coverage  was  discussed  with  survey  study  participants  during  the  interviews. 
Almost  all  of  the  study  participants  indicated  that  the  coverage  achieved  with  SPMEI  is 
good.  Most  of  the  study  participants  had  years  of  experience  in  software  development 
projects. 

Emory  (1980)  indicated  that  the  detennination  of  content  validity  is 
judgmental.  The  researcher’s  judgment  is  important.  In  this  research,  in  addition  to  the 
judgment  of  the  researcher,  many  practitioners  were  also  consulted  in  detennining  the 
content  validity  of  measurement  instrument. 

b.  Criterion-Related  Validity 

Criterion-related  validity  is  concerned  with  the  success  of  the  measures 
used  for  empirical  estimating  purposes.  As  discussed  earlier,  a  measurement  activity  is 
conducted  for  two  purposes.  These  two  purposes  are  assessment  and  prediction.  The  goal 
of  this  study  is  to  develop  a  metric  just  for  assessment  purposes  and  not  for  prediction 
purposes.  It  is  possible  to  extend  the  PME  measurement  to  be  used  in  a  prediction  or 
estimation  model.  However,  that  is  outside  the  scope  of  this  work.  Most  criterion-related 
validity  concerns  are  not  applicable  in  this  study. 
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c.  Construct  Validity 

Morisio  et  al.  (2002)  defined  the  construct  validity  as  follows:  “Construct 
validity  considers  whether  the  metrics  and  models  used  in  a  study  are  valid  abstraction  of 
the  real  world  under  study.” 

In  very  simple  terms,  trying  to  answer  the  following  question  leads  the 
way  to  establish  the  construct  validity:  are  you  measuring  what  you  intend  to  measure? 
The  following  is  an  example  in  which  the  construct  validity  is  not  satisfied.  A  researcher 
intends  to  measure  the  weight  of  a  person  but  instead  the  researcher  measures  the  height 
of  the  person.  The  given  example  was  an  extreme.  Establishing  construct  validity  in  most 
studies  is  not  easy.  It  mostly  relies  on  the  judgment  of  the  researcher  to  provide  relevant 
evidence.  While  evaluating  construct  validity,  the  correlation  between  the  measurement 
results  and  another  variable  that  is  known  to  correlate  with  the  measures  is  analyzed. 
There  should  be  an  expectance  of  theoretical  background  between  the  concept  measured 
and  the  known  variable.  When  the  correlations  are  within  the  expectations,  the 
researchers  decide  whether  the  construct  validity  concerns  are  satisfied  or  not. 

In  this  research,  software  project  management  effectiveness  is  being 
measured.  Then,  the  correlation  between  this  set  of  metric  and  the  set  of  success  ratings 
of  software  projects  are  analyzed.  The  expectance  derived  from  the  literature  on  the 
subject  is  that  effectiveness  of  project  management  is  a  factor  in  achieving  project 
success.  For  example,  Capers  Jones  (2004)  expresses  that  effectiveness  in  many  project 
management  related  areas,  such  as  project  planning,  project  estimating,  change 
management,  etc.,  is  a  success  factor  for  large  software  projects,  while  ineffectiveness  in 
these  areas  is  a  failure  factor.  The  correlation  between  the  software  project  management 
effectiveness  (PME)  metric  and  project  success  rating  was  found  to  be  strong.  This  is  one 
of  the  most  important  pieces  of  evidence  in  this  research  supporting  the  construct  validity 
of  the  instrument.  Furthermore,  most  project  management  areas  measured  with  the 
instrument  highly  correlate  with  other  areas  as  well  as  the  PME  metric.  These  high 
correlations  provide  evidence  for  the  existence  of  construct  validity  in  this  measurement 
study. 
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3.  Reliability  of  the  Measurement 


Reliability  is  about  the  consistency  of  the  measurement  results.  Emory  (1980) 
states  that  a  measure  is  reliable  to  the  degree  that  it  provides  consistent  results.  The 
expectance  in  a  reliable  measurement  tool  is  that  it  is  free  from  random  or  unstable  errors. 
Reliability  and  validity  are  related.  A  valid  tool  is  expected  to  be  reliable  as  well. 
However,  a  reliable  tool  may  not  be  valid.  A  reliable  measurement  may  provide 
consistent  results,  but  the  validity  concerns  may  not  necessarily  be  satisfied  for  that 
measurement  activity.  For  example,  the  measurement  tool  may  not  be  measuring  what  it 
is  intended  to  measure;  but  on  the  other  hand,  while  it  is  measuring  something  else  it  may 
still  produce  reliable  results. 

Understanding  reliability  requires  multiple  measurements  of  the  same  thing  that  is 
being  measured.  If  multiple  measurements  are  taken  using  the  same  measurement 
instrument  with  the  same  study  participant  and  the  results  are  consistent,  then  the 
measurement  is  considered  stable,  which  is  one  aspect  of  reliability.  This  is  particularly 
hard  in  some  surveys  because  of  the  test-retest  concerns.  The  study  participants  will 
repeat  the  answers  without  thinking  when  the  time  intervals  are  short.  In  this  research, 
stability  of  the  measurement  could  not  be  investigated  mainly  because  of  this  test-retest 
concern.  Another  aspect  of  reliability  is  the  equivalence.  Emory  (1980)  states  that 
equivalence  considers  how  much  error  may  be  introduced  by  different  investigators  or  a 
different  sample  of  items  being  studied.  The  errors  resulting  from  using  different 
investigators  or  observers  are  not  a  concern  in  this  study  because  this  measurement  study 
is  standardized  in  such  a  way  that  different  investigators  will  make  absolutely  no 
difference.  The  study  participants  completed  the  SPMEI,  and  then  the  responses  were  fed 
into  the  software  project  management  evaluation  model  (SPMEM).  The  result  was  the 
software  project  management  effectiveness  (PME)  metric.  Using  different  samples  of 
items  is  another  approach  to  detennine  the  equivalence  of  the  measurement  activity.  In 
order  to  achieve  this,  alternative  or  parallel  tests  should  be  administered.  Then,  the  results 
of  each  test  must  be  compared.  The  complexity  of  measuring  project  management 
effectiveness  in  software  projects  is  so  high  that  developing  another  instrument  requires 
another  doctoral  study. 
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In  this  study,  the  reliability  of  the  instrument  is  not  analyzed  in  detail  because  of 
the  difficulties  in  establishing  it.  However,  the  instrument  is  assumed  reliable  since  its 
validity  is  established  with  relevant  evidence.  Furthermore,  PME  measures  are  rounded 
to  integers  because  exact  measures  with  high  reliability  are  not  a  primary  concern  in  this 
study. 


4.  Practicality  of  the  Measurement 

Practicality  of  the  measurement  is  a  natural  and  obvious  requirement.  Without 
achieving  a  certain  level  of  practicality,  a  reliable  and  valid  measurement  tool  will  not  be 
used.  Therefore,  ensuring  the  practicality  of  the  measurement  tool  is  important. 

PME  measurement  is  practical.  SPMEI  is  designed  as  a  self-administered  tool. 
Thus,  the  measurement  does  not  require  a  specialist  or  a  researcher.  During  interviews, 
study  participants  indicated  that  SPMEI  makes  sense  and  it  is  easy  to  understand  and 
respond  to.  In  2-3  hours,  PME  measurement  can  be  completed.  In  comparison  to  hiring  a 
consultant  to  do  the  same  assessment,  PME  measurement  using  SPMEI  and  SPMEM  is 
faster  and  cheaper. 
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IX.  FINDINGS 


The  most  important  finding  of  this  research  is  the  relationship  between  software 
project  success  and  project  management  effectiveness  with  empirical  evidence.  There  is  a 
strong  positive  correlation  between  software  project  success  and  project  management 
effectiveness.  The  Pearson  product  moment  correlation  coefficient  between  these  two 
variables  is  0.73.  Project  management  effectiveness  accounts  for  half  of  the  variation  in 
project  success  based  on  the  data  gathered  from  survey  studies.  When  project 
management  effectiveness  is  high,  project  success  is  likely. 

As  presented  in  this  research,  it  is  possible  to  develop  a  project  management 
effectiveness  metric  using  a  questionnaire -based  instrument  and  evaluation  model.  The 
instrument,  SPMEI,  is  developed  based  on  a  theory  of  project  management  proposed  in 
this  research.  The  evaluation  model  is  developed  based  on  a  theory  of  project 
management  effectiveness  measurement,  which  uses  the  theory  of  project  management  as 
a  basis.  The  survey  study  results  provide  evidence  for  the  applicability  and  viability  of 
these  proposed  theories. 

In  this  research,  a  software  project  management  framework  is  developed.  This 
framework  is  validated  with  a  worldwide  survey  of  software  practitioners.  Furthennore, 
this  framework  is  used  as  a  basis  for  the  development  of  an  evaluation  instrument  and  the 
evaluation  model.  The  results  from  the  survey  studies  show  that  the  framework  is  valid 
and  feasible  to  be  used  in  measurement  studies. 

Studies  were  conducted  in  the  past  to  identify  the  relations  between  various 
project  management  areas  or  concepts.  One  widely  researched  topic  is  the  relation 
between  risk  management  and  project  success.  There  are  also  other  studies  that 
investigate  the  relationship  between  project  success  and  project  success  factors  such  as 
communication,  organizational  commitment,  project  manager,  project  planning,  project 
estimation,  etc.  Some  of  these  studies  are  empirical.  Others  are  based  on  the  experiences 
of  practitioners.  For  example,  Capers  Jones  (2004)  identified  that  effective  project 
planning,  effective  project  cost  estimating,  effective  project  change  management, 
effective  project  quality  control  are  success  factors  for  large  software  projects.  Vemer 
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and  Evanco  (2005)  found  that  an  above-average  project  manager  is  positively  associated 
with  software  project  success.  Reel  (1999)  states  the  importance  of  building  the  right 
team,  hence  the  importance  of  staffing  and  hiring.  The  relationship  between  teamwork 
and  communication  is  widely  researched  in  organizational  behavior  discipline.  The 
findings  indicate  strong  positive  correlation  between  communication  and  teamwork.  The 
correlation  analysis  of  these  project  management  areas  was  reported  earlier  in  this  study. 
The  findings  of  this  research  are  similar  to  the  findings  of  prior  literature  on  the  subject. 
Thus,  these  similar  findings  support  the  validity  of  this  research. 

One  interesting  discovery  of  this  study  was  finding  an  almost  perfect  positive 
correlation  between  process  area  scores  and  project  management  effectiveness  scores. 
The  Pearson  correlation  coefficient  was  0.97.  It  is  important  to  note  that  correlation  does 
not  necessarily  indicate  causation.  Therefore,  this  strong  positive  correlation  does  not 
mean  that  being  effective  in  process  related  project  management  areas  is  adequate  for 
achieving  project  success.  This  high  correlation  only  suggests  that  a  shorter  version  of  the 
measurement  activity  can  be  achieved  with  process  area  related  sections  in  SPMEI. 

The  correlations  between  main  project  management  area  scores  and  project 
success  ratings  are  all  positively  strong.  The  r  values  are  also  close  to  each  other.  This 
suggests  that  the  viability  of  the  framework  as  well  as  evaluation  model.  The  process  area 
scores  are  highly  correlated  with  the  other  three  main  area  scores.  This  may  suggest  that 
the  process  main  area  is  the  key  area  that  binds  all  other  areas.  The  correlations  between 
other  main  area  scores  are  not  as  strong  as  the  correlations  between  process  area  scores 
and  other  main  areas. 

The  mean  scores  of  risk  main  area  are  lower  than  the  means  of  other  area  scores. 
This  suggests  that  in  most  software  projects,  there  is  room  for  improvement  in  applying 
better  risk  management  practices.  For  example,  in  most  of  the  projects  critical  team 
members  do  not  have  substitutes.  Thus,  when  a  critical  team  member  leaves  the  project 
organization,  development  efforts  suffer.  In  one  case,  it  was  observed  that  such  a  loss  of  a 
critical  team  member  was  one  of  the  major  causes  of  a  significant  schedule  overrun. 
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Good  risk  management  practices  suggest  contingency  plans  in  case  of  such  losses. 
During  the  interviews,  some  study  participants  also  indicated  the  necessity  of  applying 
better  risk  management  practices. 

Different  evaluation  models  with  variations  of  different  coefficients  and  project 
management  main  areas  were  tested  based  on  the  dataset.  These  different  evaluations 
models  did  not  provide  better  results.  The  conclusion  from  these  investigations  is  that  the 
current  model  is  valid  and  adequate  for  measuring  project  management  effectiveness  in 
software  projects. 

One  important  finding  was  about  the  data  collection  challenges  faced  during  this 
study.  It  was  observed  that  getting  participation  from  software  managers  requires  a  trust 
relationship  between  the  researcher  and  study  participants.  A  portion  of  the  data  gathered 
from  projects  is  considered  sensitive  by  some  practitioners.  The  researchers  have  to  be 
aware  of  this  difficulty  and  be  careful  to  satisfy  the  expected  privacy  concerns  of  the 
study  participants. 

The  survey  studies  include  projects  developed  in  the  U.S.  and  Europe.  Only  one 
project  includes  developers  from  India,  the  United  Kingdom  and  the  U.S.  A  brief  analysis 
was  conducted  to  identify  possible  differences  among  projects  developed  in  different 
geographical  regions.  The  analysis  showed  that  there  were  no  significant  differences  in 
tenns  of  applying  project  management  principles  and  activities.  There  may  be  two 
possible  conclusions.  First,  the  cultural  issues  in  different  geographical  regions  do  no 
affect  project  management  effectiveness  as  much  as  applying  sound  project  management 
principles.  Second,  differences  might  exist,  but  these  differences  could  not  be  identified 
due  to  the  limited  sample  size  in  the  dataset.  More  research  is  required  to  shed  light  on 
this  issue.  In  addition,  more  samples  are  required  including  projects  developed  in  other 
parts  of  the  world. 

The  studies  included  projects  from  both  public  and  private  sector.  An  initial 
analysis  showed  that  there  are  no  significant  differences  between  these  two  types  of 
projects.  The  conclusion  is  that  main  project  management  principles  are  the  same  for  both 
of  these  development  environments.  Open  source  development  environments  are  quite 


249 


different  from  commercial  and  government  environments.  The  survey  studies  did  not 
include  open-source  projects.  Further  research  that  includes  open-source  projects  may 
bring  new  insight. 

Interviews  were  conducted  with  study  participants.  The  interviews  revealed  that 
the  study  participants  were  not  generally  concerned  with  the  size  of  the  application  in 
terms  of  lines  of  code.  They  were  concerned  with  whether  the  required  system 
functionality  was  delivered  or  not.  Many  project  estimations  were  based  on  the  number  of 
modules  providing  the  required  functionality.  The  author  had  the  following  observation 
during  the  studies:  lines  of  code  metrics  may  not  be  reliable  for  understanding  the  true 
development  effort.  In  some  projects,  new  technologies  may  need  to  be  developed,  or  an 
existing  technology  may  be  imported  to  a  new  hardware  platform,  or  a  significant  amount 
of  user  manuals  and  other  types  of  documentation  may  need  to  be  delivered  with  the 
project.  All  these  efforts,  which  require  a  significant  amount  of  work,  are  hard  to  capture 
with  lines  of  code  metrics.  Therefore,  these  metrics  may  be  misleading  for  understanding 
the  development  effort. 

Another  significant  finding  is  about  the  importance  of  the  communication.  The 
projects  that  achieved  high  project  management  effectiveness  conducted  weekly  or 
biweekly  meetings  with  the  inclusion  of  managers,  developers  and  other  stakeholders.  In 
addition,  it  was  possible  to  conduct  these  meetings  even  when  the  project  team  size  is 
large.  One  project  manager  published  a  project  newsletter  in  addition  to  these  project 
meetings  to  infonn  the  developers  and  other  stakeholders  about  the  status  of  the  project. 
A  new  trend  in  project  management  is  the  use  of  wiki  pages  to  facilitate  communication 
among  the  project  team  and  other  stakeholders.  In  this  research,  a  survey  on  the 
importance  of  project  management  areas  was  conducted  with  worldwide  participation. 
The  survey  results  indicate  that  communication  is  the  most  important  project 
management  area.  The  observation  from  the  survey  studies  supports  such  a  conclusion. 

The  study  participants  found  the  software  project  management  evaluation 
instrument  quite  successful  in  capturing  the  essence  of  project  management.  Most  of  the 
study  participants  specifically  indicated  that  the  coverage  with  SPMEI  was  good.  Some 
of  the  participants  mentioned  that  they  may  also  use  this  tool  as  a  checklist  to  better 
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manage  their  projects.  The  study  participants  were  asked  their  thoughts  on  the  length  of 
the  instrument.  They  indicated  that  the  length  of  the  instrument  is  what  is  necessary.  They 
were  also  asked  whether  the  instrument  should  be  shortened.  The  study  participants  think 
that  the  instrument  in  its  current  version  is  good.  It  takes  about  2-3  hours  to  complete  the 
instrument.  Most  of  the  study  participants  completed  the  instrument  within  this  time.  In 
addition,  they  indicated  that  the  instrument  made  sense  to  them.  This  is  quite  important. 
There  are  various  tools  used  by  software  managers  for  different  purposes.  When  these 
tools  do  not  make  sense  to  them  or  they  do  not  understand  the  goal  or  the  inner  workings 
of  the  tool,  they  tend  to  ignore  the  results  of  the  tool.  The  author  attended  a  seminar  on 
earned  value  management.  In  the  seminar,  the  views  and  the  beliefs  of  practitioners  in 
using  various  project  management  tools  came  up  as  an  interesting  discussion  topic.  The 
presenter  and  various  practitioners  indicated  when  these  tools  do  not  make  sense  to  the 
tool  users,  the  findings  of  the  tools  are  ignored  and  the  benefit  from  these  tools  could  not 
be  gained.  Therefore,  it  is  important  that  the  goal  of  the  tool  is  clear  and  understandable 
by  the  practitioners.  SPMEI  seems  to  satisfy  this  criterion. 

Most  of  the  projects  in  the  dataset  were  successful  software  projects.  The  mean 
project  management  effectiveness  was  high  in  this  dataset.  This  result  is  not  attributed  to 
subjectivity  of  study  participants  in  trying  to  present  their  projects  more  successful.  It  is 
observed  that  study  participants  were  objective  in  responding  to  SPMEI.  However,  it 
seems  that  most  study  participants  chose  successful  project  for  analysis. 

The  survey  studies  include  a  variety  of  software  projects  in  many  aspects. 
However,  whether  the  dataset  is  a  fair  representation  of  the  population  or  not  is  somewhat 
of  an  open  question.  Currently,  it  is  not  possible  to  identify  how  well  the  population  is 
represented  in  the  dataset  because  such  identification  requires  knowledge  about  the  state 
of  the  software  industry.  Many  questions  need  to  be  answered  about  the  software  industry 
today.  Some  of  these  questions  are: 

How  do  we  decide  whether  a  software  project  is  successful  or  not? 

What  percentage  of  software  projects  are  successful? 

What  kinds  of  projects  are  being  developed  today? 

What  are  the  types  of  software  projects? 
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What  is  the  number  of  software  projects  developed  in  the  past  and  the 
number  of  projects  being  developed  at  the  present  moment? 

What  is  the  average  schedule  of  software  projects? 

What  are  the  trends? 

Some  software  engineering  scholars  are  calling  for  standardization  across  the 
software  industry  in  collecting  basic  software  project  data.  Without  the  necessary 
standardization  and  common  understanding,  it  is  hard  to  provide  answers  to  these 
questions. 

A  traditional  view  of  project  success  is  insufficient.  That  traditional  view  of 
success  is  about  completing  the  project  on  time,  within  budget  and  with  the  necessary 
functionality.  A  few  study  participants  indicated  that  priorities  are  different  from  one 
project  to  another.  For  example,  one  study  participant  indicated  that  the  cost  was  not  an 
issue  or  a  priority  for  the  project  he  participated  in.  The  main  priority  was  on  delivering 
the  necessary  functionality  because  the  organization’s  continuity  depended  on  the 
successful  deployment  of  the  system  developed.  When  the  cost  of  the  project  is  not  a 
priority,  evaluating  the  success  of  the  project  based  on  cost  performance  may  be 
misleading.  During  the  interviews,  one  senior  software  practitioner  who  has  more  than 
thirty  years  of  industry  experience  indicated  an  important  point.  The  success  of  a  project 
relies  on  successfully  managing  expectations.  When  the  project  satisfies  the  expectations, 
then  it  should  be  considered  successful.  The  expectations  on  the  outcome  of  a  project 
should  be  negotiated  with  stakeholders.  Even  though  a  project  is  categorized  as  a  success 
based  on  the  traditional  criteria,  it  may  not  be  viewed  as  a  success  by  all  stakeholders. 
The  opposite  case  is  also  possible. 

Glass  (1999)  emphasizes  the  need  for  a  new  theory  of  project  success.  There  are 
doctoral  studies  investigating  new  views  of  software  project  success.  Some  of  them  are 
discussed  in  prior  sections.  In  the  beginning  of  this  research,  the  author  considered  the 
development  of  a  project  success  metric  using  the  traditional  criteria.  During  the  research, 
it  became  clear  that  these  criteria  are  not  sufficient  to  detennine  whether  a  project  is  a 
success  or  a  failure.  Therefore,  the  author  chose  to  ask  the  study  participants  to  rate  the 
success  of  the  project.  This  method  is  also  used  in  prior  studies.  With  the  current  state  of 
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art,  it  is  one  of  the  most  reliable  methods  in  determining  project  success.  In  determining 
project  success,  many  questions  may  be  asked  on  the  outcome  of  project.  Some  of  these 
are: 

Was  the  project  developed  on  time,  within  budget  and  with  the  necessary 
functionality? 

Was  the  customer  satisfied  with  the  system? 

Did  the  project  yield  a  good  return  on  investment? 

Did  the  project  team  members  learn  new  things? 

Did  the  organization  developing  the  project  gain  an  increased  market 
share? 

Did  the  organization  developing  the  project  gain  more  reputation? 

Did  the  project  become  an  innovation  to  the  society? 
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X.  CONCLUSIONS  AND  FUTURE  DIRECTIONS 


A.  CONCLUSIONS 

In  the  recent  years,  society  has  seen  enormous  improvements  in  the  computing 
field.  Many  devices  that  were  dreams  from  the  past  have  become  common  gadgets  in  our 
daily  lives  today.  These  devices  include  laptops,  cell  phones,  electronic  identification 
cards,  thumb  drives,  GPS  devices,  small  multimedia  players,  and  many  more.  The 
computing  power  of  CPUs  has  increased  exponentially.  The  memory  devices  are  small 
and  cheap.  Wireless  networking  enables  connection  among  many  devices.  The  software 
engineering  discipline  has  also  advanced  with  the  introduction  of  new  programming 
languages  such  as  Java,  new  design  languages  such  as  UML,  new  automated  tools  such 
as  automatic  verification  validation  tools,  etc.  Even  though  there  have  been  so  many 
technical  advances,  software  projects  still  suffer  from  significant  failures  when  compared 
to  other  project  based  industries.  In  order  to  increase  the  rate  of  software  project 
successes,  we  need  to  do  better  in  software  project  management.  The  advances  in 
software  project  management  are  slow.  Most  of  the  commonly  used  project  management 
tools  such  as  Gantt  charts,  Pert  charts,  CPM  analysis,  earned  value  management  (EVM), 
and  work  breakdown  structure  (WBS)  were  developed  decades  ago.  Gantt  charts  were 
developed  back  in  1910.  Today,  most  of  the  automated  project  management  tools  include 
these  classic  tools.  Advancements  in  software  project  management  will  help  to  increase 
the  rate  of  software  project  successes. 

In  this  research,  a  novel  project  management  tool  is  introduced.  That  tool  is  a 
software  project  management  effectiveness  metric.  This  metric  will  be  an  important 
addition  to  the  current  set  of  project  management  tools.  This  tool  measures  the  project 
management  effectiveness  in  software  projects.  This  measurement  tool  will  help  software 
development  managers  to  evaluate,  monitor  and  improve  project  management 
effectiveness  in  software  projects.  Without  an  understanding  and  assessment  of  the  status, 
it  is  hard  to  justify  the  amount  of  improvement  gained  in  any  process  improvement  effort. 
This  tool  will  help  to  improve  project  management  practices  in  software  projects.  Thus,  it 
will  assist  software  development  managers  to  achieve  better  outcomes  in  their  projects. 
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Most  of  the  literature  on  software  project  management  is  based  on  practitioner 
experience  reports.  Empirical  studies  are  not  nearly  where  they  should  be.  Conducting 
project  management  research  is  challenging.  Software  companies  are  reluctant  to  provide 
project  data  for  obvious  reasons.  In  addition,  the  studies  provide  data  that  is  hard  to 
compare  because  of  different  research  settings  under  various  assumptions.  Capers  Jones 
calls  for  standardization  in  gathering  and  reporting  software  project  data  in  his  book  titled 
“Software  Assessments,  Benchmarks,  and  Best  Practices.”  Because  of  the  nature  of  this 
study,  a  standard  comprehensive  software  project  data  collection  instrument  is  developed. 
An  important  contribution  of  this  study  is  the  introduction  of  this  data  collection 
instrument  for  other  researchers  to  gather  data  on  project  management  practices  and 
areas.  This  will  help  other  researchers  in  their  studies. 

One  of  the  most  important  contributions  of  this  research  is  the  identification  of  the 
empirical  relation  between  software  project  success  and  project  management 
effectiveness.  Prior  studies  established  that  effective  project  management  is  a  success 
factor  in  software  projects.  However,  a  clear  empirical  relation  was  lacking.  The  findings 
of  this  study  indicate  that  half  of  the  variation  in  project  success  can  be  attributed  to 
effective  project  management.  Some  of  the  other  success  factors  are  choosing  the  right 
problem  to  be  solved,  customer  and  user  satisfaction,  market  share  gain,  and  return  on 
investment  from  the  project. 

In  this  research,  a  theory  of  project  management  was  introduced.  This  theory  is 
simple  and  provides  a  fresh  view  on  the  essence  of  project  management.  This  theory  was 
developed  because  of  an  obvious  need.  At  the  beginning  of  this  research,  the  author 
struggled  in  capturing  the  essence  of  project  management  in  a  simple  way.  The  theory  of 
project  management  was  the  starting  point  for  this  research.  The  development  of  the 
theory  contributed  to  further  understanding  and  guidance  toward  the  right  direction.  The 
existence  of  this  research  is  evidence  for  the  applicability  of  this  theory.  With  the 
guidance  of  the  theory  of  project  management,  this  research  was  successfully  completed. 
In  addition,  a  theory  of  project  management  effectiveness  measurement  was  developed 
based  on  the  theory  of  project  management.  This  shows  that  the  theory  of  project 
management  is  capable  of  providing  new  insights  for  development  of  other  theories  and 
advancing  the  state  of  art. 
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The  core  concepts  in  the  theory  are  activities  and  entities.  In  simple  tenns, 
measurement  of  project  management  effectiveness  relies  on  detennining  the  effectiveness 
of  project  management  related  activities  and  entities.  The  identification  of  what  these 
activities  and  entities  are  is  guided  by  the  software  project  management  framework 
proposed  in  this  study.  This  framework  is  simple  and  inclusive.  Achieving  simplicity  in 
the  theory  and  the  framework  was  important.  Since  the  nature  of  project  management  is 
complex,  development  of  an  instrument  to  measure  its  effectiveness  is  a  challenging  task. 
Developing  a  simple  theory  and  a  simple  framework  significantly  helped  to  reduce  this 
complexity.  This  software  project  management  framework  was  validated  with  a 
worldwide  survey  of  software  practitioners.  The  results  of  this  survey  also  indicated  that 
there  has  not  been  a  change  in  the  trend  of  the  challenges  faced  in  software  development 
efforts.  Software  projects  are  still  challenged  in  areas  such  as  scope  management, 
requirements  management,  project  planning  and  estimation,  communication,  etc. 

The  main  finding  of  this  study  indicates  that  it  is  possible  to  measure  software 
project  management  effectiveness  with  the  metric  proposed  in  this  research.  Furthermore, 
during  the  interviews,  the  study  participants  indicated  that  software  practitioners  would 
benefit  from  this  project  management  effectiveness  metric  and  use  it  to  guide  their 
project  development  efforts. 

Quality  research  enables  researchers  to  ask  many  new  questions  while  providing 
answers  to  the  old  ones.  This  research  opens  up  many  possibilities  for  future  studies. 
Researchers  will  be  able  to  analyze  and  possibly  quantify  the  effects  of  software  project 
management  practices  effectiveness  to  project  success  using  the  metric  proposed  here. 

The  contributions  of  this  research  are  briefly  outlined  in  Table  52. 
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Contribution 

Contributions  to  the  Software  Engineering 

Body  of  Knowledge 

1 .  Introduction  of  a  theory  of 

project  management 

-  Provides  a  new  perspective  for  program  and 

project  management  with  a  focus  on  core 

concepts;  activities  and  entities 

-  Enables  researchers  to  conduct  further  studies 

and  develop  other  theories  on  software  projects 

and  software  project  management 

2.  Introduction  of  a  theory  of 

project  management  effectiveness 

measurement 

Enables  development  of  various  project 

management  metrics  to  guide  process 

improvement  efforts  in  software  projects 

3.  Identification  of  approaches  for 

measuring  project  management 

effectiveness 

-  Provides  directions  for  the  development  of 

other  project  management  effectiveness  metrics 

-  When  other  project  management  metrics  are 

developed,  it  provides  a  framework  for 

comparison 

4.  Development  of  a  software 

project  management  framework 

Provides  a  simple  view  for  project 

management 

-  Helps  to  focus  the  improvement  efforts  on  the 

necessary  project  management  main  areas 

-  Identifies  measurement  model  components  for 

developing  project  management  metrics 

5.  Development  of  a  self- 

evaluation  instrument  for 

software  project  management 

-  Provides  a  standardized  tool  for  collecting  data 

from  software  projects  for  conducting  project 

management  research 

-  This  instrument  may  be  used  as  a  project 

management  checklist  by  software  managers 
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-  The  instrument  may  be  used  for  developing 

new  industry  standards  and  supplementing 

existing  ones 

6.  Development  of  a  metric  for 

software  project  management 

effectiveness 

-  Enables  quantification  of  project  management 

effectiveness  in  software  projects 

-  Helps  the  software  project  managers  to  better 

manage  their  projects 

-  Helps  organizations  to  determine  whether  a 

project  requires  cancellation 

-  Helps  organizations  to  identify  the  project 

management  effectiveness  in  comparison  to 

industry 

-  Helps  to  analyze  effective  and  ineffective 

project  management  practices  in  software 

projects 

-  May  be  used  as  an  input  for  project  estimation 

purposes 

-  Provides  a  project  monitoring  tool  to  software 

project  managers 

Table  52.  Contributions  to  Defense  Community  and  Software  Engineering  Body  of 

Knowledge 
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B.  FUTURE  DIRECTIONS 

1.  Development  of  an  Automated  Tool 

Currently,  the  measurement  activity  is  conducted  manually.  The  software  project 
management  evaluation  instrument,  SPMEI,  is  provided  to  study  participants  as  a 
Microsoft  Word  document.  The  study  participants  fill  out  the  instrument.  Then  the  author 
feeds  the  data  into  an  Excel  spreadsheet,  which  contains  the  evaluation  model.  The  metric 
is  also  computed  by  hand  to  double-check  the  results.  After  all  these  tasks  are  completed, 
the  responses  are  checked  for  consistency  and  validity.  The  overall  process  takes  about  4- 
6  hours.  An  automated  tool  will  significantly  reduce  the  time  to  complete  the  process. 

Development  of  an  automated  tool  has  additional  benefits.  It  helps  to  store  and 
compare  the  measurement  results.  When  an  organization  uses  the  automated  tool  for 
measuring  the  project  management  effectiveness  in  its  projects  in  a  continuous  fashion, 
the  organization  has  a  record  of  project  management  effectiveness  history.  The  trends  in 
project  management  effectiveness  may  be  analyzed.  The  cost  effective  activities  for 
achieving  higher  effectiveness  in  project  management  will  be  identified.  Process 
improvement  efforts  may  be  initiated  based  on  the  results  of  these  measurement 
activities.  The  results  from  the  measurement  activity  will  provide  sound  justifications, 
and  these  justifications  guide  better  process  improvement  efforts. 

This  automated  tool  may  evolve  into  an  expert  system.  This  system  may  help 
project  managers  in  their  decision-making  in  providing  cost-effective  solutions  to  their 
project  management  challenges.  However,  development  of  an  expert  system  requires 
additional  research  and  more  data  from  projects. 

The  tool  automates  data  collection  efforts  for  researchers.  Best  project 
management  practices  will  be  researched  easier  with  the  help  of  the  tool. 
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2.  Increasing  the  Sample  Size 


The  sample  size  in  this  study  was  limited.  Conducting  more  studies  provides 
further  insight  to  the  applicability  and  limitations  of  the  proposed  metric.  The  studies 
include  projects  from  the  U.S.  and  Europe.  Studies  on  projects  developed  in  other  parts  of 
the  world  may  reveal  new  insights. 

An  important  issue  needs  to  be  addressed  here.  While  conducting  more  studies  on 
software  projects  helps  to  understand  the  applicability  and  limitations  of  the  metric,  it 
will  not  help  to  identify  whether  the  sample  is  a  good  representation  of  the  population 
because  many  important  data  about  the  population  of  software  projects  is  lacking. 

Advanced  statistical  analysis  methods  could  not  be  applied.  These  advanced 
statistical  methods  such  as  various  tests  of  significance  require  established  or  at  least 
estimated  knowledge  about  the  population.  Software  project  management  literature  lacks 
the  empirical  data  needed  to  establish  sufficient  knowledge  about  the  overall  population 
of  software  projects.  Such  knowledge  includes  the  categorization  of  software  project 
development  efforts,  the  success  and  failure  rates  in  these  different  categories,  empirical 
data  on  the  practices  and  methods  applied  in  these  efforts,  the  effectiveness  and 
efficiency  of  these  practices  and  efforts,  and  the  different  distributions  of  all  these  data. 
For  example,  one  of  the  basic  and  most  important  data  about  software  project 
management  is  the  rate  of  software  project  successes  and  failures.  Yet,  we  lack  that  data. 
Different  studies  yield  different  results,  especially  about  the  success  or  failure  rates  in  IT 
software  projects  (The  Standish  Group,  1994;  El  Emam  and  Kora,  2008).  Various  authors 
discussed  this  topic  in  detail  (Glass,  2002;  Glass,  2005;  Jorgensen  and  Molokken- 
Ostwold,  2006;  Emam  and  Kora,  2008).  El  Emam  and  Kora  (2008)  indicate  that,  “So, 
the  software  community  still  needs  a  reliable  global  estimate  of  software  project 
cancellation  rates  that  will  help  us  detennine  whether  there  is  a  software  crisis.” 

3.  Conducting  an  In-depth  Analysis  of  Project  Management  Areas  based 
on  the  Data  Gathered  with  SPMEI 

The  data  gathered  with  SPMEI  is  quite  extensive.  SPMEI  is  one  of  the  most 
comprehensive  project  management  data  collection  tools  developed.  This  instrument 


261 


gathers  data  from  fifteen  project  management  areas  and  over  500  data  points.  This  data  is 
used  to  evaluate  the  project  management  effectiveness  of  a  software  project.  This  data 
may  also  be  used  to  analyze  the  relationships  of  project  management  areas.  It  is  possible 
to  conduct  research  on  the  common,  best  or  worst  project  management  practices  using  the 
data  collected  with  SPMEI. 

The  sections  of  the  instrument  may  be  used  as  a  standard  data  collection  tool.  The 
strong  positive  correlations  between  various  project  management  areas  and  the  similar 
findings  in  literature  indicate  the  high  quality  of  the  instrument.  The  sections  in  the 
instrument  may  be  improved  or  adjusted  for  specific  research  goals. 

SPMEI  is  designed  as  a  modular  instrument.  SPMEM  is  a  modular  evaluation 
model  as  well.  Future  work  may  include  improvement  of  specific  portions  of  these  tools. 
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APPENDIX  A:  GLOSSARY 


Communication 

It  is  the  exchange  of  ideas,  opinions  and  information 

through  written  or  spoken  words,  symbols  or  actions. 

Configuration  Management 

A  discipline  applying  technical  and  administrative 

direction  and  surveillance  to  (1)  identify  and  document 

the  functional  and  physical  characteristics  of  a 

configuration  item,  (2)  control  changes  to  those 

characteristics,  (3)  record  and  report  change  processing 

and  implementation  status,  and  (4)  verify  compliance 

with  specified  requirements. 

Leadership 

The  ability  to  lead,  including  inspiring  others  in  a 

shared  vision.  Leaders  have  clear  visions  and  they 

communicate  these  visions  to  their  employees.  They 

foster  an  environment  within  their  companies  that 

encourages  risk  taking,  recognition  and  rewards,  and 

empowerment  allowing  other  leaders  to  emerge. 

Organizational  Commitment 

Organizational  commitment  is  the  employee’s 

psychological  attachment  to  the  organization  and 

organizational  goals. 

Process 

A  sequence  of  steps  perfonned  for  a  given  purpose;  for 

example  the  software  development  process  (IEEE, 

1990). 

Project  Monitoring  & 

Control 

Project  monitoring  is  the  process  of  keeping  the  project 

and  project  related  factors  under  observation.  Project 

control  is  to  ensure  that  project  goes  according  to  what 

is  planned  and  deviations  from  the  plan  kept  under 

control. 
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Project  Planning/Estimation 

Project  planning  is  the  process  to  quantify  the  amount 

of  time  and  budget  a  project  will  cost.  The  purpose  of 

project  planning  is  creating  a  project  plan  that  a  project 

manager  can  use  to  track  the  progress  of  his  team. 

Estimation  includes  creating  estimates  of  project  cost 

and  schedule  using  various  tools  and  techniques. 

Quality  Engineering 

In  engineering,  quality  control  and  quality  engineering 

are  involved  in  developing  systems  to  ensure  products 

or  services  are  designed  and  produced  to  meet  or 

exceed  customer  requirements.  It  involves  all  activities 

and  commitment  towards  development  of  a  high 

quality  product  to  meet  or  increase  the  customer/user 

satisfaction. 

Requirements  Management 

The  management  of  all  requirements  received  by  or 

generated  by  the  project,  including  both  technical  and 

nontechnical  requirements  as  well  as  those 

requirements  levied  on  the  project  by  the  organization. 

Risk  Assessment 

A  process  or  a  set  of  activities  that  involves 

measurement  of  risks  to  determine  priorities  and  to 

enable  identification  of  appropriate  level  of  risk 

treatment. 

Risk  Control 

That  part  of  risk  management  which  involves  the 

implementation  of  policies,  standards,  procedures  and 

physical  changes  to  eliminate  or  minimize  adverse 

risks. 

Scope  Management 

Scope  management  is  the  process  of  keeping  track  of 

scope  changes  and  limiting  the  changes  to  the  point 

that  they  are  not  disruptive  to  the  success  of  the  project. 
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Software  Project 

Management  Effectiveness 

Metric  (Software  PME 

Metric) 

This  metric  is  a  measure  of  the  project  management 

effectiveness  in  a  software  project.  It  captures  the 

effectiveness  of  the  project  management  from  the  start 

of  the  project  to  the  customer  delivery. 

Staffing  &  Hiring 

Staffing  is  the  practice  of  finding,  evaluating,  and 

establishing  a  working  relationship  with  future 

colleagues  on  a  project  and  firing  them  when  they  are 

no  longer  needed.  Staffing  involves  finding  people, 

who  may  be  hired  or  already  working  for  the  company 

(organization)  or  may  be  working  for  competing 

companies. 

Stakeholder  Involvement 

Stakeholder  involvement  is  the  early  and  extensive 

engagement  of  stakeholders  in  the  process  of  planning, 

decision  making,  and  implementation  of  a  project. 

Supplementary  Activities 

Supplementary  activities  are  activities  conducted  which 

are  not  directly  related  to  the  project  outcome. 

However,  these  activities  indirectly  increase  the 

success  probability  of  the  project.  Such  activities 

include  use  of  project  management,  development, 

testing  and  other  types  of  tools,  training  of  the 

personnel,  logistics,  increasing  the  satisfaction  of  the 

work  environment  etc. 

Teamwork 

Teamwork  is  the  concept  of  people  working  together 

towards  a  common  goal  set  as  a  team. 

Technical  Complexity 

Technical  complexity  refers  to  the  complexity  of  the 

design,  product,  project  deliverables  and  technologies 

used  in  the  development  of  the  product. 
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APPENDIX  B:  PILOT  SURVEY  STUDY:  SELF-ADMINISTERED 

QUESTIONNAIRE 


Dear  Colleague, 

Your  participation  in  this  survey  will  help  improve  the  body  of  knowledge  to  better 
manage  software  projects.  You  will  remain  anonymous.  If  you  have  questions  about  the 
survey  or  the  research,  or  if  you  want  to  have  the  results  and  the  analysis  of  the  survey, 
please  send  an  e-mail  to  kdemir@nps.edu.  We  sincerely  appreciate  your  participation  in 
this  survey. 

The  Purpose  of  the  Survey:  The  goal  of  this  survey  is  to  analyze  software  project 
management  practices  and  principles.  The  survey  results  will  help  to  determine  the 
rankings  between  various  concepts  in  software  project  management.  The  survey  takes 
about  10-15  minutes  to  complete. 

Contact  Information: 

Kadir  Demir  Software  Engineering 
PhD  Candidate 

Computer  Science  Department, 

Naval  Postgraduate  School,  Monterey,  CA,  93940 

Tel:  1-831-394-3199 

Fax:  1-831-394-3199 

Email:  kdemir@nps.edu 

Web:  http://www.nps.navy.mil/cs/kadirdemir/ 
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PARTICIPANT  CONSENT  FORM  &  MINIMAL  RISK  CONSENT  STATEMENT 


Introduction:  You  are  invited  to  participate  in  a  study  of  software  project  management 
practices.  With  infonnation  gathered  from  you  and  other  participants,  we  hope  to 
discover  insight  on  the  importance  of  certain  practices.  We  ask  you  to  read  this  form  and 
clicking  'yes'  to  the  question  below  indicates  that  you  agree  to  be  in  the  study. 

Procedures:  If  you  agree  to  participate  in  this  study,  you  will  be  asked  to  complete  a 
survey  instrumentation  composed  of  a  set  of  questions. 

Risks  and  Benefits:  I  understand  that  this  project  does  not  involve  greater  than  minimal 
risk  and  involves  no  known  reasonably  foreseeable  risks  or  hazards  greater  than  those 
encountered  in  everyday  life.  I  have  also  been  informed  of  any  benefits  to  myself  or  to 
others  that  may  reasonably  be  expected  as  a  result  of  this  research. 

Compensation:  I  understand  that  no  tangible  reward  will  be  given.  I  understand  that  a 
copy  of  the  research  results  will  be  available  at  the  conclusion  of  the  survey  research 
upon  my  request. 

Confidentiality  &  Privacy  Act:  The  records  of  this  study  will  be  kept  confidential.  No 
information  will  be  publicly  accessible  which  could  identify  you  as  a  participant. 

Voluntary  Nature  of  the  Study:  I  understand  that  my  participation  is  strictly  voluntary, 
and  if  I  agree  to  participate,  I  am  free  to  withdraw  at  any  time  without  prejudice.  A  copy 
of  this  form  will  be  provided  upon  request  for  your  records. 

Points  of  Contact:  If  you  have  any  further  questions  or  comments  after  the  completion  of 
the  study,  you  may  contact  the  research  supervisor,  James  Bret  Michael  (831)  656-2655, 
bmichael@nps.edu,  or  the  researcher,  Kadir  Alpaslan  Demir  (83 1)  394-3199, 
kdemir@nps.edu. 

Statement  of  Consent:  By  entering  my  name  and  my  signature  to  this  form,  I  am 
acknowledging  that  I  have  read  and  understood  this  information  and  agree  to 
voluntarily  participate  in  this  survey.  I  also  understand  that  I  may  stop  at  any  time 
upon  my  request. 


Name: 


Signature: 


268 


Software  Project  Management  Survey  Part  1 


1.  Please  indicate  your  roles  and  corresponding  experience  in  software  projects,  (in 
Years) 


Project  Manager 
Requirement  Engineer 
Software  Designer 
Software  Maintenance 
Software  System  Engineer 
Other 


Project  Team  Leader 
Software  Architect 
Software  Tester 
Software  Code  Developer 
Researcher/Scientist 


Software  Project  Management  Survey  Part  2 


2.  How  would  you  rate  the  importance  of  a  particular  concept,  practice  or  role 
within  software  project  management? 


Risk  Control  0  1 

Project  Monitoring  /  Control  0  1 

Communication  0  1 

Requirements  Management  0  1 

Project  Planning/Estimation  0  1 

Leadership  0  1 

Teamwork  0  1 

Staffing/Hiring  0  1 

Stakeholder  Involvement  0  1 

Organizational  Commitment  0  1 

Scopc/Con figuration  Man.  0  1 

Quality  Engineering  0  1 

Project  Manager  0  1 

Risk  Assessment  0  1 

Support  Activities  (Tools,  0  1 


Training,  Work  Environment  etc.) 


2  3  4  5 
2  3  4  5 
2  3  4  5 
2  3  4  5 
2  3  4  5 
2  3  4  5 
2  3  4  5 
2  3  4  5 
2  3  4  5 
2  3  4  5 
2  3  4  5 
2  3  4  5 
2  3  4  5 
2  3  4  5 
2  3  4  5 


6  N/O 

6  N/O 

6  N/O 

6  N/O 

6  N/O 

6  N/O 

6  N/O 

6  N/O 

6  N/O 

6  N/O 

6  N/O 

6  N/O 

6  N/O 

6  N/O 

6  N/O 
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Software  Project  Management  Survey  Part  3 


Four  different  areas  have  been  identified  for  project  management.  In  this  section,  we 
want  you  to  rate  these  areas  with  a  percentage  regarding  their  importance  within  software 
project  management.  These  are  people,  product,  process  and  risk. 

3.  The  total  rating  should  add  up  to  %100. 

*  People  related  concepts  and  practices  (Project  Manager,  Staffing/Hiring,  Leadership, 
Communication,  Teamwork,  Stakeholder  Involvement,  Organizational  Commitment) 

*  Product  related  concepts  and  practices  (Quality  Engineering,  Requirement  Engineering) 

*  Process  related  concepts  and  practices  (Project  Planning/Estimation, 
Scope/Configuration  Management,  Project  Monitoring  and  Control,  Support  Activities 
(training,  tools  etc.) 

*  Risk  related  concepts  and  practices  (Risk  Assessment,  Risk  Control) 


Please  use  (0,1 0,20...,  100) 

People  related  concepts  and  practices  . . . .% 

Process  related  concepts  and  practices  ....% 

Product  related  concepts  and  practices  . . . .% 

Risk  related  concepts  and  practices  ....% 


Total  =  100% 
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Software  Project  Management  Survey  Part  4 

This  section  is  about  your  views  and  ideas  about  the  importance  of  software  project 
management  principles  and  practices.  It  is  open-ended. 

4.  According  to  you,  what  are  the  most  important  principles  and  practices  in 
software  project  management? 


5.  According  to  you,  is  there  an  area,  activity,  concern  or  dimension  that  is  left  out 
in  this  survey? 


6.  How  can  this  survey  be  improved? 


Thank  you!!! 

If  you  have  questions  about  the  survey  or  the  research,  or  if  you  want  to  have  the  results 
and  the  analysis  of  the  survey,  please  send  an  e-mail  to  kdemir@nps.edu. 

Research  Contact  Information: 

Kadir  Demir 

Software  Engineering  PhD  Candidate 
Computer  Science  Department, 

Naval  Postgraduate  School,  Monterey,  CA,  93940 
Tel:  1-831-394-3199 
Fax:  1-831-394-3199 

Email:  kdemir@nps.edu  Web:  http://www.nps.navy.mil/cs/kadirdemir/ 

THANK  YOU  FOR  YOUR  CONSIDERATION/PARTICIPATION. 
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APPENDIX  C:  SURVEY  INSTRUMENT:  SELF-ADMINISTERED 

QUESTIONNAIRE 


Dear  Colleague, 

Your  participation  in  this  survey  will  help  improve  the  body  of  knowledge  to  better  manage 
software  projects.  You  will  remain  anonymous.  If  you  have  questions  about  the  survey  or  the 
research,  or  if  you  would  like  the  results  and  the  analysis  of  the  survey,  please  send  an  e-mail  to 
kdemir@nps.edu. 

We  sincerely  appreciate  your  participation  in  this  survey. 

The  Purpose  of  the  Survey:  The  goal  of  this  survey  is  to  analyze  software  project  management 
concepts.  The  survey  results  will  help  to  determine  what  constitutes  as  crucial  concepts  in 
software  project  management.  The  survey  takes  about  15  minutes  to  complete. 

Contact  Information: 

Kadir  Demir 

Software  Engineering  PhD  Candidate 
Computer  Science  Department, 

Naval  Postgraduate  School,  Monterey,  CA,  93940 

Tel:  1-831-394-3199 

Fax:  1-831-394-3199 

Email:  kdemir@nps.edu 

Web:  http://www.nps.navy.mil/cs/kadirdemir/ 
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PARTICIPANT  CONSENT  FORM  &  MINIMAL  RISK  CONSENT  STATEMENT 


Introduction:  You  are  invited  to  participate  in  a  study  of  software  project  management  practices. 
With  information  gathered  from  you  and  other  participants,  we  hope  to  discover  insight  on  the 
importance  of  certain  practices.  We  ask  you  to  read  this  form  and  click  'yes'  to  the  question  below 
to  indicate  that  you  agree  to  be  in  the  study. 

Procedures:  If  you  agree  to  participate  in  this  study,  you  will  be  asked  to  complete  a  survey 
instrumentation  composed  of  a  set  of  questions. 

Risks  and  Benefits:  I  understand  that  this  project  does  not  involve  greater  than  minimal  risk  and 
involves  no  known  reasonably  foreseeable  risks  or  hazards  greater  than  those  encountered  in 
everyday  life.  I  have  also  been  informed  of  any  benefits  to  myself  or  to  others  that  may 
reasonably  be  expected  as  a  result  of  this  research. 

Compensation:  I  understand  that  no  tangible  reward  will  be  given.  I  understand  that  a  copy  of  the 
research  results  will  be  available  at  the  conclusion  of  the  survey  research  upon  my  request. 

Confidentiality  &  Privacy  Act:  The  records  of  this  study  will  be  kept  confidential.  No  information 
will  be  publicly  accessible  which  could  identify  you  as  a  participant. 

Voluntary  Nature  of  the  Study:  I  understand  that  my  participation  is  strictly  voluntary,  and  if  I 
agree  to  participate,  I  am  free  to  withdraw  at  any  time  without  prejudice.  A  copy  of  this  form  will 
be  provided  upon  request  for  your  records. 

Points  of  Contact:  If  you  have  any  further  questions  or  comments  after  the  completion  of  the 
study,  you  may  contact  the  research  supervisor,  James  Bret  Michael  (831)  656-2655, 
bmichael@nps.edu,  or  the  researcher,  Kadir  Alpaslan  Demir  (831)  394-3199,  kdemir@nps.edu. 

Statement  of  Consent:  By  entering  my  name  and  my  signature  to  this  form,  I  am  acknowledging 
that  I  have  read  and  understood  this  information  and  agree  to  voluntarily  participate  in  this 
survey.  I  also  understand  that  I  may  stop  at  any  time  upon  my  request. 


Name: 


Signature: 
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3.  Please  indicate  your  roles  and  corresponding  experience  in  software  projects,  (in  Years) 

Project  Manager  : _  Project  Team  Leader  : _ 

Requirement  Engineer  : _  Software  Architect  : _ 

Software  Designer  : _  Software  Tester  : _ 

Software  Maintenance  : _  Software  Code  Developer  : _ 

Software  System  Engineer  : _  Researcher/Scientist  : _ 

Other  : _ 

4.  What  is  the  number  of  the  projects  you  participated?  _ 


5.  How  would  you  rate  the  importance  of  a  particular  concept,  practice  or  role  within 
software  project  management 

0  =  very  unimportant  1  =  unimportant  2  =  somewhat  unimportant 

3  =  neither  important  nor  unimportant  4  =  somewhat  important  5  =  important 
6  =  very  important  N/O  =  No  Opinion 


Risk  Control 

Project  Monitoring  /  Control 
Communication 
Requirements  Management 
Technical  Complexity 
Project  Planning/Estimation 
Leadership 
Teamwork 
Staffing/Hiring 
Stakeholder  Involvement 
Organizational  Commitment 
Scope  Management 
Quality  Engineering 
Project  Manager 
Risk  Assessment 
Configuration  Management 
Support  Activities  (Tools, 
Training,  Work  Environment 


0  12  3 
0  12  3 
0  12  3 
0  12  3 
0  12  3 
0  12  3 
0  12  3 
0  12  3 
0  12  3 
0  12  3 
0  12  3 
0  12  3 
0  12  3 
0  12  3 
0  12  3 
0  12  3 
0  12  3 
:•) 


4  5  6  N/O 

4  5  6  N/O 

4  5  6  N/O 

4  5  6  N/O 

4  5  6  N/O 

4  5  6  N/O 

4  5  6  N/O 

4  5  6  N/O 

4  5  6  N/O 

4  5  6  N/O 

4  5  6  N/O 

4  5  6  N/O 

4  5  6  N/O 

4  5  6  N/O 

4  5  6  N/O 

4  5  6  N/O 

4  5  6  N/O 


6.  How  many  people  were  working  in  your  LAST  software  project? 

□  1-10  □  11-100  □  1 0 1  -  or  more 


7.  What  was  the  size  of  your  LAST  software  project  in  terms  of  SLOC?  (SLOC  :  Source 
Lines  of  Code) 

I  (small)  <20,000  SLOC 

□  (middle)  20,000  SLOC  -  2  Millions  SLOC 

□  (large)  >2  Millions  SLOC 

8.  What  was  the  type  of  your  organization  in  your  LAST  project? 

□  Government  □  Commercial  □  Government- Contract 


9.  What  kind  of  an  application  was  developed  in  your  LAST  project?  (real-time  system, 
web-based,  database,  office-type  application,  operating  system  etc.) 
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10.  In  your  LAST  project,  in  which  of  these  areas  did  you  face  challenges?  (Please  select  one 
or  more.) 

□  Risk  Control 

□  Project  Monitoring/Control 

□  Communication 
0  Requirements  Management 

□  Technical  Complexity 

□  Project  Planning/Estimation 

□  Leadership 

□  Teamwork 

□  Staffmg/Hiring 

□  Stakeholder  Involvement 


11.  In  this  section,  you  are  requested  to  consider  ALL  of  your  PAST  PROJECT 
EXPERIENCES. 

Four  different  areas  have  been  identified  for  software  project  management.  We  want  you  to  rate 
these  areas  with  a  percentage  regarding  their  importance  within  software  project  management. 
These  are  people,  product,  process  and  risk. 

The  total  rating  should  add  up  to  %100. 

*  People  related  concepts  and  practices  (Project  Manager,  Staffmg/Hiring,  Leadership, 
Communication,  Teamwork,  Stakeholder  Involvement,  and  Organizational  Commitment) 

*  Product  related  concepts  and  practices  (Quality  Engineering,  Technical  Complexity,  and 
Configuration  Management) 

*  Process  related  concepts  and  practices  (Project  Planning/Estimation,  Scope  Management, 
Project  Monitoring  and  Control,  Support  Activities  (training,  tools  etc.),  Requirements 
Management) 

*  Risk  related  concepts  and  practices  (Risk  Assessment,  Risk  Control) 

Please  use  (0,5,1 0, 1 5 ... 95, 1 00) 

People  related  concepts  and  practices  ....% 

Process  related  concepts  and  practices  ....% 

Product  related  concepts  and  practices  ....% 

Risk  related  concepts  and  practices  ....% 

Total  =  100% 


□  Organizational  Commitment 

□  Scope  Management 

□  Quality  Engineering 

□  Project  Manager 

□  Risk  Assessment 

□  Configuration  Management 

□  Support  Activities  (Tools,  Training  etc.) 

□  The  last  project  was  smooth  in  every  way. 

□  Other  (Please  specify) 
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This  section  is  about  your  views  and  ideas  about  the  importance  of  software  project  management 
principles  and  practices.  It  is  open-ended  to  provide  you  the  unbounded  freedom  to  express  your 
views. 

12.  According  to  you,  what  are  the  most  important  concepts,  principles,  or  practices  in 
software  project  management? 


13.  According  to  you,  is  there  an  area,  activity,  concept  or  dimension  that  is  left  out  in  this 
survey?  Or  anything  you  would  you  like  to  add? 
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DEFINITIONS  (Alphabetical) 


COMMUNICATION:  It  is  the  exchange  of  ideas,  opinions  and  information  through  written  or 
spoken  words,  symbols  or  actions. 

CONFIGURATION  MANAGEMENT:  A  discipline  applying  technical  and  administrative 
direction  and  surveillance  to  (1)  identify  and  document  the  functional  and  physical  characteristics 
of  a  configuration  item,  (2)  control  changes  to  those  characteristics,  (3)  record  and  report  change 
processing  and  implementation  status,  and  (4)  verify  compliance  with  specified  requirements. 

LEADERSHIP:  The  ability  to  lead,  including  inspiring  others  in  a  shared  vision.  Leaders  have 
clear  visions  and  they  communicate  these  visions  to  their  employees.  They  foster  an  environment 
within  their  companies  that  encourages  risk  taking,  recognition  and  rewards,  and  empowerment 
allowing  other  leaders  to  emerge. 

ORGANIZATIONAL  COMMITMENT:  Organizational  commitment  is  the  employee's 
psychological  attachment  to  the  organization  and  organizational  goals. 

PROJECT  MANAGER:  The  person  responsible  for  planning,  directing,  controlling,  structuring, 
and  motivating  the  project.  The  project  manager  is  responsible  for  satisfying  the  customer.  In  this 
survey,  project  manager  is  considered  as  a  role  and  authority  as  well  as  incoiporating  the  personal 
traits  within  the  role. 

PROJECT  MONITORING/CONTROL:  Project  monitoring  is  the  process  of  keeping  the 
project  and  project  related  factors  under  observation.  Project  control  is  to  ensure  that  project  goes 
according  to  what  is  planned  and  deviations  from  the  plan  kept  under  control. 

PROJECT  PLANNING/ESTIMATION:  Project  planning  is  the  process  to  quantify  the  amount 
of  time  and  budget  a  project  will  cost.  The  purpose  of  project  planning  is  creating  a  project  plan 
that  a  project  manager  can  use  to  track  the  progress  of  his  team.  Estimation  includes  creating 
estimates  of  project  cost,  schedule  and  necessary  resources  using  various  tools  and  techniques. 

QUALITY  ENGINEERING:  In  engineering,  quality  control  and  quality  engineering  are 
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involved  in  developing  systems  to  ensure  products  or  services  are  designed  and  produced  to  meet 
or  exceed  customer  requirements.  It  involves  all  activities  and  commitment  towards  development 
of  a  high  quality  product  to  meet  or  increase  the  customer/user  satisfaction. 

REQUIREMENTS  MANAGEMENT :  The  management  of  all  requirements  received  by  or 
generated  by  the  project,  including  both  technical  and  nontechnical  requirements  as  well  as  those 
requirements  levied  on  the  project  by  the  organization. 

RISK  ASSESSMENT:  A  process  that  involves  measurement  of  risk  to  determine  priorities  and 
to  enable  identification  of  appropriate  level  of  risk  treatment.  In  this  survey,  risk  assessment 
includes  the  identification  of  risks. 

RISK  CONTROL:  That  part  of  risk  management  which  involves  the  implementation  of  policies, 
standards,  procedures  and  physical  changes  to  eliminate  or  minimize  adverse  risks. 

SCOPE  MANAGEMENT:  Scope  management  is  the  process  of  keeping  track  of  scope  changes 
and  limiting  the  changes  to  the  point  that  they  are  not  disruptive  to  the  success  of  the  project. 

STAFFING/HIRING:  Staffing  is  the  practice  of  finding,  evaluating,  and  establishing  a  working 
relationship  with  future  colleagues  on  a  project  and  firing  them  when  they  are  no  longer  needed. 
Staffing  involves  finding  people,  who  may  be  hired  or  already  working  for  the  company 
(organization)  or  may  be  working  for  competing  companies. 

STAKEHOLDER  INVOLVEMENT:  Stakeholder  involvement  is  the  early  and  extensive 
engagement  of  stakeholders  in  the  process  of  planning,  decision  making,  and  implementation  of  a 
project. 

SUPPLEMENTARY  ACTIVITIES:  Supplementary  activities  are  the  type  of  activities  which 
are  not  directly  related  to  the  project  outcome.  However,  these  activities  indirectly  increase  the 
success  probability  of  the  project.  Such  activities  include  use  of  project  management, 
development,  testing  and  other  types  of  tools,  training  of  the  personnel,  logistics,  increasing  the 
satisfaction  of  the  work  environment  etc. 
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TEAMWORK:  Teamwork  is  the  concept  of  people  working  together  towards  a  common  goal  set 


as  a  team. 


TECHNICAL  COMPLEXITY:  Technical  complexity  refers  to  the  complexity  of  the  design, 
product,  project  deliverables  and  technologies  used  in  the  development  of  the  product. 
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APPENDIX  D:  SURVEY  QUESTION  #5  FURTHER  ANALYSIS 


This  appendix  provides  the  further  analysis  on  the  responses  to  the  survey 
question  #5 . 

This  question  is  worded  as  follows: 

How  would  you  rate  the  importance  of  a  particular  concept,  practice  or  role  within 
software  project  management? 

The  scale  for  the  responses  are: 

0  =  very  unimportant  1  =  unimportant 

2  =  somewhat  unimportant  3  =  neither  important  nor  unimportant 

4  =  somewhat  important  5  =  important 

6  =  very  important  N/O  =  No  Opinion 
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1. 


COMMUNICATION 


Response  Count 

0  (Very  Unimportant) 

0 

1  (Unimportant) 

0 

2  (Somewhat  Unimportant) 

0 

3  (Neither  Important  Nor  Unimportant) 

0 

4  (Somewhat  Important) 

5 

5  (Important) 

14 

6  (Very  Important) 

58 

N/O  (No  Opinion) 

1 

Total  Number  of  Responses 

78 

Question  #5  -  Communication 


Responses 


Mean 

5.688311688 

Median 

6 

Standard  Deviation 

0.590711887 

Mode 

6 

T- Value 

39.93459393 

P- Value 

9.3328535E-53 

Statistically  Significant 

YES 

Critical  Test  Value 

1.991672579 
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2. 


TEAMWORK 


Response  Count 

0  (Very  Unimportant) 

0 

1  (Unimportant) 

0 

2  (Somewhat  Unimportant) 

0 

3  (Neither  Important  Nor  Unimportant) 

3 

4  (Somewhat  Important) 

7 

5  (Important) 

23 

6  (Very  Important) 

45 

N/O  (No  Opinion) 

0 

Total  Number  of  Responses 

78 

Question  #5  -  Teamwork 


45- 


- 23 - 

_ _ 7 

0  0  0  3  - 

- 1 - 1 - 1  1  r - M - 0 - 

0  1  2  3  4  5  6 

Responses 


Mean 

5.41025641 

Median 

6 

Standard  Deviation 

0.812817729 

Mode 

6 

T- Value 

26.1889074 

P- Value 

4.3208 136E-40 

Statistically  Significant 

YES 

Critical  Test  Value 

1.991254363 

60  n 

■£  50 
°  40  - 
S)  30 

C 
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0 


283 


3. 


LEADERSHIP 


Response  Count 

0  (Very  Unimportant) 

0 

1  (Unimportant) 

0 

2  (Somewhat  Unimportant) 

1 

3  (Neither  Important  Nor  Unimportant) 

2 

4  (Somewhat  Important) 

9 

5  (Important) 

25 

6  (Very  Important) 

41 

N/O  (No  Opinion) 

0 

Total  Number  of  Responses 

78 

Mean 

5.320512821 

Median 

6 

Standard  Deviation 

0.875252687 

Mode 

6 

T- Value 

23.41519726 

P- Value 

9.2534552E-37 

Statistically  Significant 

YES 

Critical  Test  Value 

1.991254363 
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4.  REQUIREMENTS  MANAGEMENT 


Response  Count 

0  (Very  Unimportant) 

0 

1  (Unimportant) 

0 

2  (Somewhat  Unimportant) 

0 

3  (Neither  Important  Nor  Unimportant) 

1 

4  (Somewhat  Important) 

7 

5  (Important) 

45 

6  (Very  Important) 

25 

N/O  (No  Opinion) 

0 

Total  Number  of  Responses 

78 

Mean 

5.205128205 

Median 

5 

Standard  Deviation 

0.651850002 

Mode 

5 

T- Value 

29.87675835 

P- Value 

4.17349559E- 

44 

Statistically  Significant 

YES 

Critical  Test  Value 

1.991254363 
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5. 


ORGANIZATIONAL  COMMITMENT 


Response  Count 

0  (Very  Unimportant) 

0 

1  (Unimportant) 

1 

2  (Somewhat  Unimportant) 

0 

3  (Neither  Important  Nor  Unimportant) 

2 

4  (Somewhat  Important) 

15 

5  (Important) 

29 

6  (Very  Important) 

31 

N/O  (No  Opinion) 

0 

Total  Number  of  Responses 

78 

Question  #5-  Organizational  Commitment 


Responses 


Mean 

5.102564103 

Median 

5 

Standard  Deviation 

0.947858058 

Mode 

6 

T- Value 

19.59084823 

P- Value 

1.2036674E-31 

Statistically  Significant 

YES 

Critical  Test  Value 

1.991254363 
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6. 


PROJECT  MANAGER 


Response  Count 

0  (Very  Unimportant) 

0 

1  (Unimportant) 

0 

2  (Somewhat  Unimportant) 

0 

3  (Neither  Important  Nor  Unimportant) 

5 

4  (Somewhat  Important) 

13 

5  (Important) 

30 

6  (Very  Important) 

30 

N/O  (No  Opinion) 

0 

Total  Number  of  Responses 

78 

Question  #5-  Project  Manager 


Responses 


Mean 

5.102564103 

Median 

5 

Standard  Deviation 

0.947858058 

Mode 

6 

T- Value 

19.59084823 

P- Value 

1.2036674E-31 

Statistically  Significant 

YES 

Critical  Test  Value 

1.991254363 
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7. 


STAKEHOLDER  INVOLVEMENT 


Response  Count 

0  (Very  Unimportant) 

0 

1  (Unimportant) 

1 

2  (Somewhat  Unimportant) 

1 

3  (Neither  Important  Nor  Unimportant) 

2 

4  (Somewhat  Important) 

14 

5  (Important) 

31 

6  (Very  Important) 

29 

N/O  (No  Opinion) 

0 

Total  Number  of  Responses 

78 

Mean 

5.051282051 

Median 

5 

Standard  Deviation 

0.992143631 

Mode 

5 

T- Value 

18.25988897 

P- Value 

1.0509756E-29 

Statistically  Significant 

YES 

Critical  Test  Value 

1.991254363 
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8. 


PROJECT  MONITORING  AND  CONTROL 


Response  Count 

0  (Very  Unimportant) 

0 

1  (Unimportant) 

1 

2  (Somewhat  Unimportant) 

1 

3  (Neither  Important  Nor  Unimportant) 

2 

4  (Somewhat  Important) 

17 

5  (Important) 

28 

6  (Very  Important) 

29 

N/O  (No  Opinion) 

0 

Total  Number  of  Responses 

78 

Mean 

5.012820513 

Median 

5 

Standard  Deviation 

1.012821567 

Mode 

6 

T- Value 

17.55170903 

P- Value 

1.23398069E- 

28 

Statistically  Significant 

YES 

Critical  Test  Value 

1.991254363 
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9. 


PROJECT  PLANNING  AND  ESTIMATION 


Response  Count 

0  (Very  Unimportant) 

0 

1  (Unimportant) 

0 

2  (Somewhat  Unimportant) 

0 

3  (Neither  Important  Nor  Unimportant) 

4 

4  (Somewhat  Important) 

18 

5  (Important) 

31 

6  (Very  Important) 

25 

N/O  (No  Opinion) 

0 

Total  Number  of  Responses 

78 

Question  #5  -  Project  Planning  &  Estimation 


£ 

3 

O 

o 


0) 

in 

£ 

O 

Q. 

in 

<u 

cc 


60 
50 
40 
30 
20 
10 
0 

0  1  2  3  4  5  6 


Responses 


Mean 

4.987179487 

Median 

5 

Standard  Deviation 

0.875252687 

Mode 

5 

T- Value 

20.05168826 

P- Value 

2.6841819E-32 

Statistically  Significant 

YES 

Critical  Test  Value 

1.991254363 
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10.  SCOPE  MANAGEMENT 


Response  Count 

0  (Very  Unimportant) 

0 

1  (Unimportant) 

0 

2  (Somewhat  Unimportant) 

3 

3  (Neither  Important  Nor  Unimportant) 

4 

4  (Somewhat  Important) 

15 

5  (Important) 

30 

6  (Very  Important) 

25 

N/O  (No  Opinion) 

1 

Total  Number  of  Responses 

78 

Question  #5  -  Scope  Management 


Responses 


Mean 

4.909090909 

Median 

5 

Standard  Deviation 

1.041024523 

Mode 

5 

T- Value 

16.09203661 

P- Value 

3.4345063E-26 

Statistically  Significant 

YES 

Critical  Test  Value 

1.991672579 
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11.  RISK  CONTROL 


Response  Count 

0  (Very  Unimportant) 

0 

1  (Unimportant) 

0 

2  (Somewhat  Unimportant) 

3 

3  (Neither  Important  Nor  Unimportant) 

2 

4  (Somewhat  Important) 

18 

5  (Important) 

35 

6  (Very  Important) 

20 

N/O  (No  Opinion) 

0 

Total  Number  of  Responses 

78 

60 

50  - 

1  1 

O  O 

CO 

esuods 

a> 

O' 

20  - 

10  - 

0  - 

Question  #5  -  Risk  Control 


-35- 


18 

0  0  3 

l  l  i 

Responses 


20 


Mean 

4.858974359 

Median 

5 

Standard  Deviation 

0.963278479 

Mode 

5 

T- Value 

17.04389474 

P- Value 

7.4926949E-28 

Statistically  Significant 

YES 

Critical  Test  Value 

1.991254363 
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12.  STAFFING  AND  HIRING 


Response  Count 

0  (Very  Unimportant) 

0 

1  (Unimportant) 

1 

2  (Somewhat  Unimportant) 

1 

3  (Neither  Important  Nor  Unimportant) 

6 

4  (Somewhat  Important) 

21 

5  (Important) 

22 

6  (Very  Important) 

26 

N/O  (No  Opinion) 

1 

Total  Number  of  Responses 

78 

Mean 

4.818181818 

Median 

5 

Standard  Deviation 

1.108902643 

Mode 

6 

T- Value 

14.38762979 

P- Value 

2.1320399E-23 

Statistically  Significant 

YES 

Critical  Test  Value 

1.991672579 
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13.  CONFIGURATION  MANAGEMENT 


Response  Count 

0  (Very  Unimportant) 

0 

1  (Unimportant) 

0 

2  (Somewhat  Unimportant) 

0 

3  (Neither  Important  Nor  Unimportant) 

8 

4  (Somewhat  Important) 

23 

5  (Important) 

23 

6  (Very  Important) 

24 

N/O  (No  Opinion) 

0 

Total  Number  of  Responses 

78 

Question  #5  -  Configuration  Management 


Responses 


Mean 

4.807692308 

Median 

5 

Standard  Deviation 

0.994239151 

Mode 

6 

T- Value 

16.05761166 

P- Value 

2.7271650E-26 

Statistically  Significant 

YES 

Critical  Test  Value 

1.991254363 
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14.  RISK  ASSESSMENT 


Response  Count 

0  (Very  Unimportant) 

0 

1  (Unimportant) 

1 

2  (Somewhat  Unimportant) 

2 

3  (Neither  Important  Nor  Unimportant) 

5 

4  (Somewhat  Important) 

22 

5  (Important) 

28 

6  (Very  Important) 

20 

N/O  (No  Opinion) 

0 

Total  Number  of  Responses 

78 

Question  #5  -  Risk  Assessment 


Responses 


Mean 

4.717948718 

Median 

5 

Standard  Deviation 

1.079892201 

Mode 

5 

T- Value 

14.05002485 

P- Value 

6.006 1044E-23 

Statistically  Significant 

YES 

Critical  Test  Value 

1.991254363 
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15.  QUALITY  ENGINEERING 


Response  Count 

0  (Very  Unimportant) 

0 

1  (Unimportant) 

1 

2  (Somewhat  Unimportant) 

3 

3  (Neither  Important  Nor  Unimportant) 

9 

4  (Somewhat  Important) 

12 

5  (Important) 

35 

6  (Very  Important) 

16 

N/O  (No  Opinion) 

2 

Total  Number  of  Responses 

78 

Mean 

4.644736842 

Median 

5 

Standard  Deviation 

1.139636753 

Mode 

5 

T- Value 

12.58162596 

P- Value 

3.6536650E-20 

Statistically  Significant 

YES 

Critical  Test  Value 

1.992102124 
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16.  SUPPORT  ACTIVITIES 


Response  Count 

0  (Very  Unimportant) 

0 

1  (Unimportant) 

2 

2  (Somewhat  Unimportant) 

2 

3  (Neither  Important  Nor  Unimportant) 

7 

4  (Somewhat  Important) 

13 

5  (Important) 

37 

6  (Very  Important) 

22 

N/O  (No  Opinion) 

0 

Total  Number  of  Responses 

78 

Mean 

4.269230769 

Median 

4 

Standard  Deviation 

1.027834401 

Mode 

4 

T- Value 

10.90598118 

P- Value 

2.7880999E-17 

Statistically  Significant 

YES 

Critical  Test  Value 

1.991254363 
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17.  TECHNICAL  COMPLEXITY 


Response  Count 

0  (Very  Unimportant) 

0 

1  (Unimportant) 

1 

2  (Somewhat  Unimportant) 

5 

3  (Neither  Important  Nor  Unimportant) 

18 

4  (Somewhat  Important) 

14 

5  (Important) 

30 

6  (Very  Important) 

7 

N/O  (No  Opinion) 

3 

Total  Number  of  Responses 

78 

Question  #5  -  Technical  Complexity 


Responses 


Mean 

4.173333333 

Median 

4 

Standard  Deviation 

1.178332919 

Mode 

5 

T- Value 

8.623509173 

P- Value 

8.5633 12  IE- 1 3 

Statistically  Significant 

YES 

Critical  Test  Value 

1.992543466 
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APPENDIX  E:  MAPPING  OF  THE  PROJECT  MANAGEMENT 
PROCESSES  TO  THE  PROJECT  MANAGEMENT  PROCESS 
GROUPS  AND  THE  KNOWLEDGE  AREAS  [FROM  (PMI,  2004)] 
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APPENDIX  F:  SOFTWARE  PROJECT  MANAGEMENT 
EVALUATION  INSTRUMENT  (SPMEI) 


Dear  Fellow  Colleague, 

I  sincerely  appreciate  for  taking  time  to  participate  in  this  study.  This  study  is 
conducted  as  part  of  my  PhD  research.  I  am  testing  the  applicability  of  a  self-evaluation 
instrument  for  software  project  management.  We  would  like  you  to  apply  the  instrument 
on  a  software  project  you  have  managed.  Your  participation  will  be  anonymous.  (Please 
get  the  3 -digit  code  by  sending  an  e-mail  to  kdemir@nps.edu  -  if  you  do  not  already  have 
one.) 

The  benefits  of  your  participation: 

It  will  result  in  a  tool  for  YOU  to  monitor,  evaluate  and  improve  YOUR  projects. 

You  will  have  private  third  party  evaluation  of  your  software  project. 

You  will  have  first-hand  access  to  research  results.  It  will  result  in  the 
development  of  project  management  metrics  and  improve  the  body  of  knowledge  to 
better  manage  and  evaluate  software  projects. 

The  only  requirement  to  participate  is: 

You  are  a  software  development  project  manager. 

Y ou  are  a  software  development  technical  manager. 

Or  you  have  worked  as  one  in  the  past. 

The  study  will  be  conducted  with  discretion  in  complete  privacy.  And  neither  will 
it  possible  to  trace  the  results  back  to  a  particular  person,  organization  or  any  entity. 

This  questionnaire  investigates  what  happened  during  a  particular  project 
development.  Any  other  use  will  definitely  be  incorrect  and  misleading. 

This  is  NOT  an  evaluation  of  the  project  manager,  the  management  team,  or  any 

other  person.  This  instrument  is  not  designed  for  that  purpose.  Any  inference  derived  for 
such  a  purpose  will  definitely  be  incorrect  and  misleading. 

301 


This  is  NOT  an  evaluation  of  the  organization.  It  focuses  on  the  project,  not  the 
organization. 

If  you  have  questions  about  the  study  or  the  research,  please  send  an  e-mail  to 
kdemir@nps.edu. 

Contact  Information: 

Kadir  Alpaslan  Demir 

Software  Engineering  PhD  Candidate, 

Computer  Science  Department, 

Naval  Postgraduate  School, 

Monterey,  CA,  93943 

Tel:  1-831-333-9277 

Fax:  1-831-333-9277 

Email:  kdemir@nps.edu 

W eb :  http  ://faculty .nps  .edu/kdemir/ 
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Informed  Consent  Form  (Naval  Postgraduate  School) 

Introduction.  You  are  invited  to  participate  in  a  study  entitled  Software  Project 
Management  Effectiveness  Evaluation. 

Procedures.  The  goal  of  this  study  is  to  test  an  evaluation  tool  to  aid  practitioners  assess 
their  software  project  managements.  You  will  be  asked  to  fill  out  a  questionnaire.  The  process 
takes  1,5-3  hours  depending  on  the  participant. 

Risks  and  Benefits.  I  understand  that  this  project  does  not  involve  greater  than  minimal 
risk  and  involves  no  known  reasonably  foreseeable  risks  or  hazards  greater  than  those 
encountered  in  everyday  life.  I  have  also  been  informed  of  any  benefits  to  myself  or  to  others 
that  may  reasonably  be  expected  as  a  result  of  this  research. 

Compensation.  I  understand  that  no  tangible  compensation  will  be  given.  I  understand 
that  a  copy  of  the  research  results  will  be  available  at  the  conclusion  of  the  experiment.  It  will  be 
delivered  to  you  in  the  method  you  find  appropriate. 

Confidentiality  &  Privacy  Act.  I  understand  that  all  records  of  this  study  will  be  kept 
confidential  and  that  my  privacy  will  be  safeguarded.  No  information  will  be  publicly  accessible 
which  could  identify  me  as  a  participant.  I  will  be  identified  only  as  a  code  number  on  all 
research  forms/data  bases.  My  name  on  any  signed  document  will  not  be  paired  with  my  code 
number  in  order  to  protect  my  identity.  I  understand  that  records  of  my  participation  will  be 
maintained  by  NPS  for  three  years,  after  which  they  will  be  destroyed. 

Voluntary  Nature  of  the  Study.  I  understand  that  my  participation  is  strictly  voluntary, 
and  if  I  agree  to  participate,  I  am  free  to  withdraw  at  any  time  without  prejudice. 

Points  of  Contact.  I  understand  that  if  I  have  any  questions  or  comments  regarding  this 
project  upon  the  completion  of  my  participation,  I  should  contact  the  Principal  Investigator,  Dr. 
John  Osmundson.  (831)656-3775,  iosmundson@nps.edu  or  Co-PI  Mr.  Kadir  Alpaslan  Demir 
(831)  333-9277,  kdemir@nps.edu  .  Any  medical  questions  should  be  addressed  to  LTC  Eric 
Morgan,  MC,  USA,  (CO,  POM  Medical  Clinic),  (831)  242-7550, 

eric.morgan@nw.amedd.army.mil.  Any  other  questions  or  concerns  may  be  addressed  to  the  IRB 
Chair,  LT  Brent  Olde,  656-3807,  baolde@nps.edu. 

Statement  of  Consent.  I  have  been  provided  with  a  full  explanation  of  the  purpose, 
procedures,  and  duration  of  my  participation  in  this  research  project.  I  understand  how  my 
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identification  will  be  safeguarded  and  have  had  all  my  questions  answered.  I  have  been  provided 
a  copy  of  this  form  for  my  records  and  I  agree  to  participate  in  this  study.  1  understand  that  by 
agreeing  to  participate  in  this  research  and  signing  this  form,  I  do  not  waive  any  of  my  legal 
rights.  Sending  the  completed  questionnaire  instrument  to  Co-PI  (Kadir  Alpaslan  Demir),  shows 
my  agreement  to  participate  in  the  study. 


Participant’s  Signature 

SIGNED  XXX _ 

Researcher’s  Signature 


4/20/2008 

Date 


Date 
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DIRECTIONS  FOR  FILLING  OUT  THE  QUESTIONNAIRE: 

There  are  16  sections  in  the  questionnaire.  The  first  section  covers  some  basic 
statistics  regarding  the  project.  The  rest  15  sections  are  organized  under  various  titles.  It 
takes  about  1,5  to  3  hours  depending  on  the  participant. 

•  Think  of  a  project  you  participated  and  have  extensive  knowledge.  The 
questionnaire  examines  from  the  start  of  the  project  (from  the  point  it  is 
decided  that  the  project  will  be  undertaken)  to  the  point  it  is  delivered  to 
the  customer  for  the  first  time  (or  it  is  cancelled). 

•  The  project  you  chose  does  not  have  to  be  a  successful  or  a  good  example. 
Our  interest  is  testing  whether  the  instrument  works  or  not.  We  are  trying 
to  provide  a  tool  for  you  that  you  may  benefit  in  your  future  projects. 

•  You  may  respond  in  any  order  you  like. 

•  The  questions  are  straightforward  and  designed  to  be  simple  and  easy  to 
understand. 

•  There  are  two  main  types  of  questions. 

In  the  first  type,  we  simply  would  like  you  to  check  one  or  more  statements  that 
apply  to  the  project. 

Check  the  STATEMENT  that  applies  to  the  project.  (CHECK  ONLY  ONE) 

□  X 

Sy 

□  None 

Check  the  STATEMENT/S  that  applies  to  the  project.  (CHECK  ALL  THAT  APPLY) 

Sx 

□  Y 

Sz 

□  None 

In  the  second  type,  we  would  like  to  get  your  opinion  whether  you  agree  or  not  on 


a  particular  statement. 


Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

SI1 

STATEMENT 

Agree 

Disagree 

Applicable 

□ 

□ 

IKI 

□ 

□ 

□ 
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•  When  there  are  combined  statements,  consider  them  as  one  concept  and 
respond  as  is,  or  take  an  average  of  the  ratings  for  each  of  the  statement. 

•  The  questionnaire  is  designed  as  a  whole.  Trying  to  infer  results  from  just 
one  or  more  sections  will  be  misleading. 

•  Please  respond  to  all  questions.  Partial  responses  will  prevent  getting  a 
successful  evaluation. 
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A.  GENERAL  PROJECT  RELATED  QUESTIONS  (17  QUESTIONS  - 
ABOUT  5  MINUTES) 

Directions:  Please  provide  responses  to  the  following  questions  to  the  best  of  your 
knowledge. 

ENTER  THE  CODE  PROVIDED: 


PR1. 

What  was  the  goal  of  the  project?  What  kind  of  an  application  was 
developed?  What  were  the  deliverables?  Please  briefly  state. 

PR2. 

What  was  the  title  of  the  project  (if  there  is  one)? 

PR3. 

What  was  the  projected/planned  effort  for  the 
project?  (in  terms  of  man-month) 

Man- 

month 

PR4. 

What  was  the  actual  effort  for  the  project? 
(in  terms  of  man-month) 

Man- 

month 

PR5. 

What  was  the  actual  cost  of  the  project? 

Dollars 

PR6. 

What  was  the  projected/planned  budget  for 
the  project? 

Dollars 

PR7. 

How  long  did  the  project  take? 

*From  start  (or  contract)  date  to  delivery 
date 

Months 

PR8. 

What  was  the  projected/planned  schedule  for 
the  project? 

Months 

PR9. 

What  was  the  start  date  of  the  project? 
(Month/Y  ear) 

/ 

PR10. 

What  was  the  delivery  date  of  the  project? 
(Month/Y  ear) 

/ 
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PR11. 

How  much  of  the  functionality  (or  number  of 
features)  are  delivered  to  the  customer? 
(Between  the  initial  baseline  and  the 
delivered  product) 

% 

PR12. 

How  many  people  did  work  on  the  project?  (Including  the  management, 
consultants/contractors  etc.) 

Requirements  Phase  : 

Design  Phase  : 

Implementation  Phase  : 

Testing  and  Delivery  Phase  : 

Total  : 

Or 

Average  number  of  people  during  from  start  to  end  : 

PR13. 

What  is  the  size  of  the  project?  (in  terms  of 
Lines  of  Code  (KLOC)  or  function  points  (FP)  ) 

KLOC 

FP 

PR14. 

Where  was  the  project  developed?  Which  state,  country  or  countries? 

PR15. 

What  kind  of  an  organization  did  develop  the  project?  (Such  as 
government,  commercial,  open  source  community,  government  contract 
etc.)  Organization  name? 

PR16. 

How  woulc 
(0  being  co 

[ 

1  you  rate  the  overall  success  of  the  f 
mplete  failure  and  10  being  the  com 

jroject? 
plete  success.) 

0 

□ 

11  2  3  4  5  6 

:□□□□□□ 

7  8  9  1 

]  □  □  □ 

PR17. 

What  is/was  your  role  in  the  project? 
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B.  COMMUNICATION  SECTION  (23  QUESTIONS  -  ABOUT  7-12 
MINUTES) 

Cl.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

I  |  A  common  glossary/terminology  for  the  project  is  created. 

H  Communication  procedures  adapts  due  to  changing  project  environment. 

I  I  Communication  procedures  are  always  followed  as  stated  in  the  communication 
planning  documentation  (or  similar  document). 

I  I  There  is  a  project  infonnation  distribution  list  (or  a  similar  document)  and  it  is 
maintained. 

I  I  The  project  budget  includes  resources  for  communication  and  project  infonnation 
distribution  efforts. 

I  I  None 

C2.  Who  are  generally  present  in  the  project  status  meetings?  (Check  all  that 
apply.) 

P  Project  manager 

[~|  Project  team  leaders 

I  |  Project  team  members 

I  I  Customer/s  and/or  user  representatives 

I  |  Various  stakeholders  or  stakeholder  representatives 

I  I  Executive  management  /  Project  sponsor 

C3.  Which  of  the  following/s  is/are  discussion  items  in  project  status  meetings? 
(Check  all  that  apply.) 

P  Project  schedule 
P  Project  budget 
|~~|  Project  risks 
1  |  Project  staff  problems 

P  Important  development  events  and/or  accomplished  project  deliverables 
P  Requirements 
I  I  None 

C4.  Which  of  the  following/s  does  the  project  information  distribution  plan/list  (or 
similar  document)  contain?  (Check  all  that  apply.) 

I  |  Project  information  type/context  (What  will  be  communicated) 

I  |  Recipients  of  various  communication  items  (Stakeholders-  who  should  receive  the 
information) 

P  Project  related  information  distribution  frequency 
P  Timeframe  of  the  relevant  communication 

I  |  Communication  fonnat  and  medium  (How  the  communication  will  be  conducted- 
reports,  meetings,  teleconferencing  etc.) 

I  |  Responsible  project  staff  for  communication 
|  |  Not  available 
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(5) 

(4) 

(3) 

(2) 

(i) 

N/A 

C5 

The  importance  of 
communication  is  understood 
and  established  between 
stakeholders  and  project  team 
members.  There  is 
commitment  to  good 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

C6 

Stakeholders  including  project 
team  members’  needs  for 
various  project  data  and 
information  are  analyzed  and 
identified. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

C7 

There  have  been 
communication  problems  due 
to  various  reasons. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

C8 

Communication  is  used  as  a 
means  to  resolve  conflicts. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

C9 

There  are  designated  project 
team  members  and 
representatives  of  stakeholders 
responsible  for  conducting 
communication. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

CIO 

Communication  procedures 
are  documented  and 
distributed  to  stakeholders 
and  project  team  members. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

Cll 

Communication  and 
coordination  for  activities  are 
planned  in  the  project  plan. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

C12 

The  response  and 
acknowledgement  procedures 
are  planned  and  documented 
in  the  communication 
procedures. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 
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C13 

The  information  needs  of 
stakeholders  and  project  team 
members  are  satisfied  in  a 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

timely  manner  through 
appropriate  use  of 
communications  media. 

□ 

□ 

□ 

□ 

□ 

□ 

As  a  project  manager  or  a 
project  team  member,  I  can 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

C14 

easily  communicate  my 
messages  and  I  can  be 
understood. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

C15 

A  communications  and  project 
information/data  management 
system  with  essential 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

capabilities  are  in  place.  (Such 
as  databases,  mail  servers,  or 
teleconferencing  etc.) 

□ 

□ 

□ 

□ 

□ 

□ 

C16 

The  project  environment 
facilitates  horizontal 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

communication  that  is  between 
peers. 

□ 

□ 

□ 

□ 

□ 

□ 

The  project  team  operates  in  a 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

C17 

virtual  environment  rather 

Agree 

□ 

□ 

□ 

Disagree 

Applicable 

than  on  a  face-to-face  basis. 

□ 

□ 

□ 

The  project  status  is  visible  to 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

C18 

every  stakeholder  and  project 
team  member. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

The  project  manager, 
management  team,  and  team 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

C19 

leaders  are  always  accessible 

Agree 

□ 

□ 

□ 

Disagree 

Applicable 

to  project  team  members  in  a 
timely  manner. 

□ 

□ 

□ 

When  I  report  a  project 
problem,  I  get  timely 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

C20 

acknowledgement  that  my 
message  has  been  received  and 
understood. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 
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C21 

Informal  communications 
within  the  team  and 
stakeholders  are  also  an 
important  part  of  project 
development  environment. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

C22 

The  project  environment 
facilitates  free-format 
meetings  for  various  purposes. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

C23 

The  project  environment 
facilitates  freedom  in 
reporting  of  project  problems. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 
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C.  TEAMWORK  SECTION  (30  QUESTIONS  -  ABOUT  10  MINUTES) 

Tl.  Which  of  the  following/s  are  clearly  documented  in  the  project  plan  for  each 
team  member?  (Check  all  that  apply.) 

|~|  Responsibility  of  the  team  member 

□  Accountability  of  the  team  member 

I  |  Authority  of  the  project  manager  and  team  members 
O  Reporting  structure 

I  I  Interfaces  and/or  communication  channels 
I  I  None 

T2.  How  many  project  team  members  stayed  with  the  project  until  the  end 
according  to  the  project  staffing  plan?  (Check  only  one.) 

□  All 

I  |  Most 
I  |  Some 
I  I  None 

T3.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

I  I  Notable  project  accomplishments/milestones/deliverables  are  celebrated  with  social 
events  or  parties. 

I  I  There  are  problem-solving  meetings  with  the  attendance  of  relevant  project  team 
members  and  stakeholders. 

I  I  Organizational  culture  encourages  problem  solving  sessions  with  the  attendance  of 
project  members. 

I  |  When  a  project  team  member  left  the  team  or  the  member  is  removed,  the  rest  of  the 
team  has  understood  the  reasoning. 

I  I  None 

T4.  Which  of  the  following  activities  are  carried  out  throughout  the  project?  (Check 
all  that  apply.) 

□  Social  events/parties 

|  |  Team  building  training 
|  |  Introduction  meetings  and  parties 
I  |  Reward  and  other  types  of  ceremonies 
I  |  Brainstorming  and  problem  solving  meetings  and  sessions 
Meetings  for  self-assessment  of  team  performance 
I  I  None 


(5) 

(4) 

(3) 

(2) 

(1) 

N/A 

T5 

The  project  is  adequately 
staffed  during  the  project 
development. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

313 


T6 

The  organization  structure 
and  responsibility/task  matrix 
are  clearly  documented  and 
provided  to  project  team 
members. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

T7 

There  are  regular  status 
meetings  to  self-assess  the 
project  team’s  performance 
and  morale. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

T8 

There  is  an  accepted  shared 
vision  for  the  project  within 
team  members. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

T9 

Team  members  are  involved  in 
the  project  planning  effort. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

T10 

Team  members  are  involved  in 
decision-making  process 
during  project  development. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

Til 

The  project  status  is  visible  to 
team  members. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

T12 

In  order  to  do  the  work 
effectively,  all  necessary 
project  data  and  information 
is  easily  accessible  to  project 
members. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

T13 

Training  opportunities  are 
created  and  made  available 
upon  need  or  at  the  request  of 
team  members. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 
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There  are  more  experienced 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

T14 

project  team  members  than 
inexperienced  team  members. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

The  project  environment 
facilitates  teaming  up 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

T15 

inexperienced  team  members 
with  the  experienced  team 
members. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

Rewards  for  achievements  are 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

T16 

handed  out  justifiably  and 
made  the  project  team  happy. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

T17 

There  is  trust  and  respect 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

among  team  members. 

□ 

□ 

□ 

□ 

□ 

□ 

The  project  team  is 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

T18 

empowered  with  adequate 
resources  to  do  their  tasks. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

T19 

The  support  from  upper 
management  or  project 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

sponsor  is  visible  to  the  project 
team. 

□ 

□ 

□ 

□ 

□ 

□ 

The  project  offers  stimulating 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

T20 

and  challenging  work  to 
project  team  members. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

The  project  environment 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

T21 

offers  professional  growth 
potential  for  team  members. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 
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The  project  suffers  from  not 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

T22 

having  enough  experienced  or 
qualified  team  members. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

T23 

Team  members  are  tasked 
based  on  their  skills, 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

capabilities,  ambitions  and 
interests. 

□ 

□ 

□ 

□ 

□ 

□ 

The  team  members  are  clear 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

T24 

about  how  their  job 

Agree 

□ 

□ 

□ 

Disagree 

Applicable 

performance  will  be  evaluated. 

□ 

□ 

□ 

T25 

The  project  team  members 
believe  that  they  have  enough 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

resources  to  accomplish  their 
jobs  successfully. 

□ 

□ 

□ 

□ 

□ 

□ 

T26 

The  orientation  procedures 
and  the  sponsors  are 
documented  and  the 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

procedures  are  followed  for 
the  team  members  joining  the 
team  later. 

□ 

□ 

□ 

□ 

□ 

□ 

Project  priorities  are  always 
made  clear  via  meetings, 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

T27 

presentations  and  memos; 
priorities  are  not  constantly 
changing. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

The  project  suffers  from  lack 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

T28 

of  communication  and 

Agree 

□ 

□ 

□ 

Disagree 

Applicable 

coordination. 

□ 

□ 

□ 

T29 

The  project  suffers  from  lack 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

of  leadership  at  various  levels. 

□ 

□ 

□ 

□ 

□ 

□ 
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The  project  team  consists  of 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

T30 

people  who  has  worked 
together  before. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 
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D.  LEADERSHIP  SECTION  (17  QUESTIONS  -  ABOUT  3-6  MINUTES) 


(5) 

(4) 

(3) 

(2) 

(1) 

N/A 

LI 

The  leaders  at  various  levels 
promote  competition  rather 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

than  coordination  within  the 
project  organization. 

□ 

□ 

□ 

□ 

□ 

□ 

L2 

The  leaders  at  various  levels 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

sets  example  for  others. 

□ 

□ 

□ 

□ 

□ 

□ 

L3 

After  the  creation  of  the 
shared  vision  for  the  project, 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

the  leaders  at  various  levels 
maintain  the  vision. 

□ 

□ 

□ 

□ 

□ 

□ 

The  leaders  at  various  levels 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

L4 

are  effective  problem-solvers 
in  technical  and  social  issues. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

The  management  protects  the 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

L5 

team  from  outside 

Agree 

□ 

□ 

□ 

Disagree 

Applicable 

interference. 

□ 

□ 

□ 

L6 

The  leaders  at  various  levels 
clearly  state  their  leadership 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

styles  upfront  with  reasons  for 
the  style. 

□ 

□ 

□ 

□ 

□ 

□ 

The  leaders  at  various  levels 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

L7 

assign  correct  tasks  to  correct 

Agree 

□ 

□ 

□ 

Disagree 

Applicable 

people. 

□ 

□ 

□ 

The  leaders  at  various  levels 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

L8 

are  respected  by  the  team 
members. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 
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The  leaders  at  various  levels 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

L9 

easily  delegates  authority 
when  necessary. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

L10 

The  leaders  at  various  levels 
observe  the  morale  of  the  staff 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

and  takes  proactive  action  to 
boost  the  morale. 

□ 

□ 

□ 

□ 

□ 

□ 

Lll 

The  project  team  suffers  from 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

coordination  problems. 

□ 

□ 

□ 

□ 

□ 

□ 

L12 

The  project  team  suffers  from 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

communication  problems. 

□ 

□ 

□ 

□ 

□ 

□ 

The  leaders  at  various  levels 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

L13 

welcome  communication  of 

Agree 

□ 

□ 

□ 

Disagree 

Applicable 

project  problems  at  any  time. 

□ 

□ 

□ 

The  leaders  at  various  levels 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

L14 

clearly  define  what  is  expected 
from  project  team  members. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

LIS 

The  project  team  members 
freely  share  their  desires, 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

wishes,  and  concerns  with 
their  leaders. 

□ 

□ 

□ 

□ 

□ 

□ 

L16 

The  leaders  at  various  handle 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

project  politics  well. 

□ 

□ 

□ 

□ 

□ 

□ 
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*  Provide  response  to  either  L17  or  L18. 

L17.  (Answer  only  if  the  project  team  mostly  consists  of  inexperienced  staff)  Check 
the  statement  that  applies  to  the  project.  (Check  only  one.) 

i  I  The  leaders  at  various  levels  have  to  make  most  decisions  and  direct  the  staff. 

I  |  The  leaders  at  various  levels  make  most  decisions  with  the  consultation  of  team 
members  and  coach  the  staff. 

|  |  The  leaders  at  various  levels  and  the  team  members  make  decisions  together. 

I  I  The  leaders  at  various  levels  mostly  oversee  the  decisions  made  by  the  staff  and 
delegate  the  tasks. 

L18.  (Answer  only  if  the  project  team  mostly  consists  of  experienced  staff)  Check 
the  statement  that  applies  to  the  project.  (Check  only  one.) 

I  I  The  leaders  at  various  levels  have  to  make  most  decisions  and  direct  the  staff. 

I  |  The  leaders  at  various  levels  make  most  decisions  with  the  consultation  of  team 
members  and  coach  the  staff. 

r~|  The  leaders  at  various  levels  and  the  team  members  make  decisions  together. 

I  I  The  leaders  at  various  levels  mostly  oversee  the  decisions  made  by  the  staff  and 
delegate  the  tasks. 
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E.  ORGANIZATIONAL  COMMITMENT  SECTION  (27  QUESTIONS  - 
ABOUT  7-12  MINUTES) 


(5) 

(4) 

(3) 

(2) 

(1) 

N/A 

The  executive  management  is 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

OC1 

committed  to  providing 
necessary  financial  support. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

OC2 

The  executive  management  is 
committed  to  providing 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

necessary  flexibility  on  the 
project  schedule. 

□ 

□ 

□ 

□ 

□ 

□ 

The  executive  management  is 
committed  to  providing 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

OC3 

necessary  flexibility  on  the 
project  functionality  and 
quality. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

The  executive  management 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

OC4 

and  project  organization  is 
open  to  change/adaptation. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

OC5 

There  is  encouragement  for 
organizational  and  personal 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

certifications  such  as  CMMI, 
PMI,  PMP,  ISO  etc. 

□ 

□ 

□ 

□ 

□ 

□ 

OC6 

There  is  commitment  to 
quality  by  executive 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

management,  team  members 
and  other  stakeholders. 

□ 

□ 

□ 

□ 

□ 

□ 

Adequate  resources  are  set 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

OC7 

aside  for  the  success  of  the 

Agree 

□ 

□ 

□ 

Disagree 

Applicable 

project. 

□ 

□ 

□ 

OC8 

There  is  support  for  bringing 
in  expertise  when  needed 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

(Such  as  technical,  legal, 
contracting  etc.) 

□ 

□ 

□ 

□ 

□ 

□ 
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OC9 

There  is  support  for  quality 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

subcontracting  when  needed. 

□ 

□ 

□ 

□ 

□ 

□ 

OCIO 

The  executive  management 
supports  /  empowers  /  enables 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

the  project  manager  to  do  his 
job. 

□ 

□ 

□ 

□ 

□ 

□ 

There  is  continuous  and 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

OC11 

observable  support  from 

Agree 

□ 

□ 

□ 

Disagree 

Applicable 

executive  management. 

□ 

□ 

□ 

Leaders  at  various  levels  are 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

OC12 

committed  to  the  success  of  the 

Agree 

□ 

□ 

□ 

Disagree 

Applicable 

project. 

□ 

□ 

□ 

Leaders  at  various  levels  are 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

OC13 

committed  to  their  team 

Agree 

□ 

□ 

□ 

Disagree 

Applicable 

members. 

□ 

□ 

□ 

OC14 

The  project  manager  and 
leaders  at  various  levels  are 
committed  to  providing 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

continuous  support  in 
enabling  the  team  members  to 
do  their  work. 

□ 

□ 

□ 

□ 

□ 

□ 

The  project  team  members  are 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

OC15 

committed  to  the 

Agree 

□ 

□ 

□ 

Disagree 

Applicable 

accomplishment  of  the  project. 

□ 

□ 

□ 

OC16 

The  project  team  members 
show  their  commitment  to 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

staying  with  the  project  until 
the  end. 

□ 

□ 

□ 

□ 

□ 

□ 
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The  project  team  members  put 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

OC17 

extra  effort  for  the  success  of 

Agree 

□ 

□ 

□ 

Disagree 

Applicable 

the  project. 

□ 

□ 

□ 

OC18 

The  project  team  members 
lack  motivation  due  to  various 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

reasons  including  external 
factors. 

□ 

□ 

□ 

□ 

□ 

□ 

OC19 

The  project  manager  and  the 
team  members  don't  consider 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

the  project  as  a  pleasant 
challenge. 

□ 

□ 

□ 

□ 

□ 

□ 

OC20 

The  project  manager  and  the 
team  members  consider  the 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

project  as  a  valuable  learning 
experience. 

□ 

□ 

□ 

□ 

□ 

□ 

OC21 

There  is  a  friendly-work 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

environment. 

□ 

□ 

□ 

□ 

□ 

□ 

The  project  team  members 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

OC22 

publicly  and  explicitly  indicate 
their  job  satisfaction. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

There  is  commitment  from 
various  stakeholders  including 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

OC23 

project  team  members, 

Agree 

□ 

□ 

□ 

Disagree 

Applicable 

customer,  marketing  and  sales 
department(if  applicable)  etc. 

□ 

□ 

□ 

OC24 

Executive  management, 
project  manager  and  project 
team  members  are  committed 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

to  establishing  effective 
project  management  and 
control  mechanisms. 

□ 

□ 

□ 

□ 

□ 

□ 
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OC25.  Which  of  the  following  item/s  does  the  executive  management  show 
commitment  to  providing  support?  (Check  all  that  apply.) 

1~~|  Human  resources 
1  I  Training  needs 

H  Supplementary  needs  such  as  office  space,  tools,  computer  systems  etc. 

I  I  None 

0C26.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

I  |  The  executive  management  clearly  defines  the  authority  and  responsibility  of  the 
project  manager. 

□  The  executive  management  allows  for  realistic  budget  and  schedule. 

1  I  Training  is  made  available  to  all  team  members. 

I  |  There  are  some  resignations  in  the  project  organization. 

Q  The  project  organization  allows  for  career  development. 

I  I  None 
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F.  PROJECT  MANAGER  SECTION  (27  QUESTIONS  -  ABOUT  5-9 
MINUTES) 

PM1.  How  many  project  managers  have  changed  during  the  project  (Turnover)? 
(Check  only  one.) 

I  I  None 

□  l 

□  2 

□  3  or  more 

PM2.  How  many  years  of  experience  does  the  project  manager  have?  (Check  only 
one.) 

j  I  Less  Than  5 

□  5-10 

□  10-15 

□  15-20 

I  |  More  Than  20 

PM3.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

1  I  The  project  manager  has  certification  related  to  project  management  such  as  PMP  etc. 

I  |  The  project  manager  has  worked  on  similar  projects. 

I  |  The  project  manager  has  worked  as  a  project  manager  before. 

I  |  The  project  manager  has  worked  as  a  practitioner/developer  before,  therefore  has 
technical  background. 

I  |  The  project  manager  has  worked  on  different  types  of  projects. 

I  I  None 

PM4.  Which  of  the  following/s  the  project  manager  has  control  over?  (Check  all 
that  apply.) 

□  Budget 

□  Schedule 

1  I  Product  Quality 
I  |  Process  Quality 
I  |  Hiring  and  letting  go 
I  I  None 


(5) 

(4) 

(3) 

(2) 

(1) 

N/A 

PM5 

The  project  manager’s  role, 
accountability,  and 
responsibilities  are  clearly 
defined  and  communicated  to 
stakeholders  including  project 
team  members. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 
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PM6 

The  project  manager  was 
given  adequate  authority  and 
control  over  the  project. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PM7 

The  project  manager  has 
adequate  project  management 
education,  training  and 
experience. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PM8 

As  a  project  manager,  I  have 
goals  and  a  clear  vision  related 
to  the  project.  /As  a  team 
member,  I  observe  that  the 
project  manager  has  goals  and 
a  clear  vision  related  to  the 
project. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PM9 

As  a  project  manager,  I  am 
able  to  maintain  the  continuity 
of  the  project  vision.  /  Asa 
team  member,  I  observe  that 
the  project  manager  is  able  to 
maintain  the  continuity  of  the 
project  vision. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PM10 

As  a  project  manager,  I  am 
deeply  committed  to  the 
project./As  a  team  member,  I 
observe  the  deep  commitment 
in  the  project  manager. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 
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PM11 

As  a  project  manager,  I  am 
communicative  and  always 
accessible  to  team./As  a  team 
member,  I  observe  that  the 
project  manager  is 
communicative  and  always 
accessible  to  the  team. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PM12 

As  a  project  manager,  I 
motivate  staff  and  other 
people  well./As  a  team 
member,  I  observe  that  the 
project  manager  motivates  the 
staff  and  other  people  well. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PM13 

As  a  project  manager,  I  am  a 
good  planner  and 
organizer./As  a  team  member, 

I  observe  that  the  project 
manager  is  a  good  planner 
and  organizer. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PM14 

As  a  project  manager,  I  am  an 
effective  problem  solver./ As  a 
team  member,  I  observe  that 
the  project  manager  is  an 
effective  problem  solver. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PM15 

As  a  project  manager,  I 
consult  to  and  get  advice  from 
stakeholders  and  project  team 
members.  / 1  observe  that  the 
project  manager  consults  to 
and  gets  advice  from 
stakeholders  and  project  team 
members. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 
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PM16 

As  a  project  manager,  I 
delegate  easily  when 
necessary  ./As  a  team  member, 

I  observe  that  the  project 
manager  delegates  easily  when 
necessary. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PM17 

As  a  project  manager,  I  use 
rewarding  and  punishment 
mechanisms  effectively.  /As  a 
team  member,  I  observe  that 
the  project  manager  uses 
rewarding  and  punishment 
mechanisms  effectively. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PM18 

As  a  project  manager,  I  am  a 
people  person./As  a  team 
member,  I  observe  that  the 
project  manager  is  a  people 
person. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PM19 

As  a  project  manager,  I  am  an 
effective  team  builder  and 
player./As  a  team  member,  I 
observe  that  the  project 
manager  is  an  effective  team 
builder  and  player. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PM20 

As  a  project  manager,  I 
support  my  team  members  in 
various  aspects./ As  a  team 
member,  I  observe  that  the 
project  manager  supports  the 
team  members  in  various 
aspects. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 
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PM21 

As  a  project  manager,  I 
monitor  every  aspect  of  the 
project./As  a  team  member,  I 
observe  that  the  project 
manager  monitors  every 
aspect  of  the  project. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PM22 

As  a  project  manager,  I 
inform  the  stakeholders  and 
my  team  members  well./ As  a 
team  member,  I  observe  that 
the  project  manager  informs 
the  stakeholders  and  the  team 
members  well. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PM23 

As  a  project  manager,  I  clarify 
when  the  stakeholders  and  the 
team  members  are  confused 
about  an  aspect  of  the 
project./As  a  team  member,  I 
observe  that  the  project 
manager  clarifies  when  the 
stakeholders  and  the  team 
members  are  confused  about 
an  aspect  of  the  project. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PM24 

As  a  project  manager,  I  am 
able  to  see  the  project  as  a 
whole./As  a  team  member,  I 
observe  that  the  project 
manager  sees  the  project  as  a 
whole. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PM25 

As  a  project  manager,  I 
understand  the  domain  of  the 
project./As  a  team  member,  I 
observe  that  the  project 
manager  understands  the 
domain  of  the  project. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 
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PM26 

As  a  project  manager,  I 
protect  my  team  members  so 
that  their  work  don’t  get 
disrupted./As  a  team  member, 

I  observe  that  the  project 
manager  protects  us  so  that 
our  work  don’t  get  disrupted. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PM27 

As  a  project  manager,  I 
understand  and  foresee  the 
project  risks./As  a  team 
member,  I  observe  that  the 
project  manager  understands 
and  foresees  the  project  risks. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 
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G.  REQUIREMENTS  MANAGEMENT  SECTION  (27  QUESTIONS  - 
ABOUT  5-9  MINUTES) 

RM1.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

I  |  There  is  a  requirements  development  document  (how  they  are  gathered  and 
developed). 

i  I  There  is  a  requirements  management  document  (how  they  are  handled). 

[Q  There  is  an  agreed/negotiated  requirements  baseline. 

f~l  There  is  a  requirements  baseline  document  and  it  is  managed. 

I  I  None 

RM2.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

i'~l  Oral  requirements  are  used. 

I  |  Written  requirements  are  used. 

I  |  Requirements  are  formal  -  a  standard  guides  the  development;  have  identifiers  and 
traceability  matrix  etc. 

l  ~~|  Requirements  are  informal  -  requirements  are  just  identified  and  listed. 

I  I  None 

RM3.  Which  of  the  following  activities  are  conducted  in  the  project?  (Check  all  that 
apply.) 

I  |  Market  surveys 

1  |  Customer/User  interviews 

I  I  Prototyping 

I  I  Scenarios/  use  cases 

[  |  Observation  of  the  user  in  operation 

I  I  None 

RM4.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

LH  Stakeholders  are  identified  prior  to  requirements  development  activities. 

Q  Requirements  related  documents  have  versions. 

I  |  There  is  a  requirements  traceability  matrix  (or  a  similar  document  to  trace  the 
requirements  during  all  the  development  activities). 

I  |  Requirements  volatility  (number  of  requirements  change/  percent  of  number  of 
requirements  change  etc.)  metrics  are  collected  and  used. 

I  |  Testing  team  is  involved  in  the  requirement  development  activities. 

I  I  None 


(5) 

(4) 

(3) 

(2) 

(1) 

N/A 

RM5 

Requirements  prioritization  is 
conducted  and  used  for 
development  decisions. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 
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All  stakeholders  are  involved 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

RM6 

in  the  requirements 
development. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

Users  or  user  representatives 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

RM7 

are  involved  in  the 

Agree 

□ 

□ 

□ 

Disagree 

Applicable 

requirements  development. 

□ 

□ 

□ 

RM8 

Stakeholders  show 
commitment  to  requirements 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

stability  during  the  project 
development. 

□ 

□ 

□ 

□ 

□ 

□ 

Automated  requirements 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

RM9 

development  and  management 
tools  are  used. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

RM 

All  requirements  are 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

10 

traceable. 

□ 

□ 

□ 

□ 

□ 

□ 

Product  components  and 

RM 

project  deliverables  can  be 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

11 

mapped  to  specific 
requirements. 

□ 

□ 

□ 

□ 

□ 

□ 

RM 

Requirements  are  clear  / 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

12 

unambiguous. 

□ 

□ 

□ 

□ 

□ 

□ 

RM 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

IV.1t  1 

13 

Requirements  are  complete. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 
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RM 

14 

There  are  no  inconsistencies 
among  requirements. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

RM 

15 

During  the  project 
development,  requirements 
related  issues  are  resolved 
with  the  negotiation  with  the 
customers. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

RM 

16 

Requirements  are  validated 
with  the  user,  customer  and 
necessary  stakeholders. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

RM 

17 

There  are  designated  points  of 
contact  (people)  representing 
various  stakeholders  to  resolve 
requirements  related  issues. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

RM 

18 

The  procedures  are  formal  for 
requirements  validation  (what 
the  customer  want). 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

RM 

19 

The  procedures  are  formal  for 
requirements  verification  (the 
system  does  what 

requirements  state). 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

RM 

20 

There  is  a  formal 

requirements  change 

procedure  and  document. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

RM 

21 

Requirements  history  and 
rationale  for  requirements 
changes  are  recorded. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 
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RM 

Requirements  are  worded 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

IV.1t  1 

22 

simple  and  each  requirement 
consists  of  only  one  concept. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

RM 

Extra  effort  is  spent  to  make 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

23 

the  requirements  testable. 

□ 

□ 

□ 

□ 

□ 

□ 

RM 

24 

There  are  testing  plans  to 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

check  if  the  requirements  are 
implemented  as  intended. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

RM 

User/customer  profiles  are 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

25 

identified  and  documented. 

□ 

□ 

□ 

□ 

□ 

□ 

RM 

26 

Requirements  are  constantly 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

changing  and  all  changes  are 
being  implemented. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

RM 

Requirements  are  kept  stable 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

27 

at  some  point. 

□ 

□ 

□ 

□ 

□ 

□ 
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H.  STAKEHOLDER  INVOLVEMENT  SECTION  (12-16  QUESTIONS  - 
ABOUT  3-7  MINUTES) 


(5) 

(4) 

(3) 

(2) 

(1) 

N/A 

Various  users  and/or 
customers  are  involved  in  the 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

SI1 

requirements  development 
and  functionality/feature 
identification  process. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

SI2 

Various  user  and/or  customer 
concerns  are  specified  and 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

documented  for  the  project 
and  the  product. 

□ 

□ 

□ 

□ 

□ 

□ 

Various  user  and/or  customer 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

SI3 

profiles  are  identified  and 

Agree 

□ 

□ 

□ 

Disagree 

Applicable 

documented. 

□ 

□ 

□ 

SI4 

Prototypes/user  stories/paper 
mock-ups/use  cases  etc.  are 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

prepared  with  the  involvement 
of  users. 

□ 

□ 

□ 

□ 

□ 

□ 

Executive/upper  management 
is  involved  in  the  decision 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

SI5 

making  process  regarding  the 
project  baselines,  cost  and 
schedule  variations  etc. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

SI6 

All  stakeholders  are  identified 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

and  documented. 

□ 

□ 

□ 

□ 

□ 

□ 

SI7 

There  are  regular  meetings 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

with  various  stakeholders. 

□ 

□ 

□ 

□ 

□ 

□ 

SI8 

There  is  an  information 
gathering  activity  to  identify 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

stakeholders  and  their 

stakes/concerns. 

□ 

□ 

□ 

□ 

□ 

□ 
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All  stakeholders  show 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

SI9 

commitment  to  the  successful 
outcome  of  the  project. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

SI10.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

I  |  There  is  a  document  guiding  the  management  of  stakeholders. 

I  |  The  stakeholder  management  plan/document  lists  the  primary  and  secondary 
stakeholders. 

I  I  The  stakeholder  management  plan/document  lists  the  concerns  and  stakes  of  the 
primary  and  secondary  stakeholders. 

I  I  The  stakeholder  management  plan/document  provides  specific  strategies  for  dealing 
with  various  stakeholders. 

I  |  The  users  and/or  customers  participated  in  the  testing  phase  of  the  project. 

I  |  There  is  a  documented  procedure  for  the  acceptance  of  the  project  deliverables. 

I  I  None 

*  Respond  the  following  questions(SIll-SI12)  only  if  the  project  is  developed  for  the 
market  without  a  specific  contract. 


Sill 

The  marketing  department 
and  necessary  functional 
managers  are  involved  in  the 
decision  making  process 
during  development. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

SI12 

The  marketing  department 
provides  timely  information 
regarding  users  and  other 
competing  products. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

*  Respond  the  following  questions  (SI13-SI18)  only  if  the  project  is  developed  under 
a  contract  with  a  specific  customer. 


There  are  communication  and 

SI13 

coordination  problems 

between  project  team 

members  and  other 

stakeholders. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

336 


SI14 

When  there  is  a  change  in  the 
baseline,  the  cost,  schedule, 
and  functionality/features  are 
renegotiated  with  the 

customer. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

SI15 

Regular  updates  regarding 
project  variables  such  as  cost, 
schedule  and  progress  on 
functionality  are  provided  to 
the  stakeholders. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

SI16 

When  there  is  an  increase  in 
cost  or  delay  in  schedule,  the 
news  and  the  consequences  are 
shared  with  the  stakeholders 
in  a  timely  manner. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

SI17 

Project  milestones  are 
considered  reached  when 
there  is  consensus  from 
stakeholders  for  advancing  to 
the  next  phase. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

SI18.  Check  the  statement  that  applies  to  the  project.  (Check  only  one.) 

I  I  Project  team  members  are  allowed  to  have  direct  communication  with  the  customers 
and/or  users. 

I  I  All  communication  with  the  stakeholders  is  conducted  via  the  project  manager  and/or 
management. 
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I.  PROJECT  MONITORING  AND  CONTROL  SECTION(19  QUESTIONS  - 
ABOUT  4-8  MINUTES) 

PMC1.  Check  the  statement  that  applies  to  the  project.  (Check  only  one.) 

[~|  There  is  a  documented  project  plan. 

I  I  There  is  no  project  plan. 

PMC2.  Which  of  the  following  data  and/or  metric/s  are  regularly  monitored  and 
documented?  (Check  all  that  apply.) 

FI  Team/developer  perfonnance 

I  |  Cost  and  earned  value 

1  |  Risk  items  and  their  impacts 

I  I  Schedule  performance 

I  |  Number  of  requirements  changes 

r~|  Necessary  staff  and  skill  requirements 

I  I  None 

PMC3.  Check  the  statement  that  applies  to  the  project.  (Check  only  one.) 

I  |  There  are  specific  project  team  members  assigned  for  controlling  activities  such  as 
configuration  management,  requirement  changes  etc. 

I  |  All  control  activities  are  handled  by  the  project  manager. 

PMC4.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

8  I  There  are  project  progress  or  milestone  review  meetings. 

I  |  Key  project  problems  are  identified  and  being  monitored. 

I  I  Key  project  problems  and  project  progress  status  is  visible  to  the  stakeholders 
including  project  team  members. 

I  I  None 

PMC5.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

1  I  There  is  an  established  requirements  change  and  control  process. 

1  |  There  is  an  established  risk  management  and  control  process. 

I  |  There  is  an  established  configuration  management  process. 

|  |  There  is  an  established  baseline  tracking  and  scope  change  control  process. 

I  |  There  is  an  established  project  management  data  and  metrics  collection  and 
monitoring  process. 

None 


(5) 

(4) 

(3) 

(2) 

(1) 

N/A 

PMC 

6 

The  project  problems  are 
generally  proactively 
addressed  (before  they 
happen). 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 
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PMC 

7 

The  project  problems  are 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

generally  reactively  addressed 
(when  they  happen). 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

PMC 

The  project  resources  are 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

8 

closely  monitored. 

□ 

□ 

□ 

□ 

□ 

□ 

There  is  an  established  project 

PMC 

monitoring  and  control 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

9 

procedure  with  the  acceptance 
of  project  team  members. 

□ 

□ 

□ 

□ 

□ 

□ 

There  are  established 

PMC 

methods/criteria  to  determine 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

10 

deviations  from  the  project 
plan. 

□ 

□ 

□ 

□ 

□ 

□ 

PMC 

In  case  of  deviations  from  the 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

plan,  corrective  action  is  taken 

Agree 

□ 

□ 

□ 

Disagree 

Applicable 

11 

immediately. 

□ 

□ 

□ 

Project  management  metrics 
are  effectively  collected  and 

PMC 

used  in  decision-making,  (such 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

12 

as  planned  versus  actual  cost, 
requirements  changes, 
schedule  performance  etc.) 

□ 

□ 

□ 

□ 

□ 

□ 

A  project  management 

PMC 

automated  software  tool  is 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

13 

used  to  manage  project 
management  data  and  metrics. 

□ 

□ 

□ 

□ 

□ 

□ 

PMC 

Earned  value  management  is 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

14 

effectively  used. 

□ 

□ 

□ 

□ 

□ 

□ 
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PMC 

15 

There  is  communication 
between  management  and 
project  staff  regarding  the 
project  progress  data. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PMC 

16 

The  commitment  and  concerns 
of  various  stakeholders  is 
being  monitored  through 
regular  meetings  and 
communication. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PMC 

17 

The  subcontractor 
performance  is  monitored 
regularly. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PMC 

18 

There  are  checklists  for 
critical  tasks  such  testing, 
version  control,  requirements 
change  requests  etc. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PMC 

19 

Corrective  actions  for 
problems  are  timely  and 
effective. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

340 


J.  PROJECT  PLANNING  AND  ESTIMATION  SECTION  (35  QUESTIONS  - 
ABOUT  10-18  MINUTES) 

PPE1.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

j  |  There  is  a  formal  documented  project  plan. 

□  There  is  an  informal  project  plan. 

I  I  There  project  plan  and  schedule  is  made  visual  via  diagrams,  charts  etc. 

I  I  There  is  no  project  plan. 

PPE2.  Check  the  statement  that  applies  to  the  project.  (Check  only  one.) 

Q  The  project  plan  is  developed  as  needed  during  the  project. 

I  I  The  project  plan  is  developed  up  front  before  any  development  effort. 

PPE3.  Check  the  statement  that  applies  to  the  project.  (Check  only  one.) 

I  I  The  project  budget,  schedule,  and  staff  requirements  are  strictly  enforced  by  the 
executive/upper  management  or  customer. 

I  I  The  project  budget,  schedule,  and  staff  requirements  are  identified  via  analysis  and 
negotiation. 

PPE4.  Check  the  statement  that  applies  to  the  project.  (Check  only  one.) 

I  I  The  project  plan  is  approved  by  the  stakeholders  such  as  customers,  users,  project 
team  members,  executive  management  etc. 

I  I  There  is  no  approval  process. 

PPE5.  Which  of  the  following/s  is/are  involved  in  the  project  planning?  (Check  all 
that  apply.) 

1  ~|  Senior/executive/upper  management 
O  Experts  and  consultants 
I  |  Project  manager  and/or  management  team 
Q  Project  team  members 
Q  Customer/user/marketing  department 
O  Other  relevant  stakeholders 
I  I  None 
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PPE6.  Which  of  the  following/s  is/are  included  in  the  project  plan?  (Check  all  that 
apply.) 

1~|  Project  scope 
1  I  Deliverables  or  products  list 

i  I  Detailed  schedule  and  milestones  /  various  product  version  delivery  dates 

n  Detailed  budget  and  cost  analysis 

PI  Staffing/personnel/developer  requirements 

I  I  Task  responsibility  matrix  or  similar  assignment  matrix 

I  I  Required  functionality/features  of  the  products  or  deliverables 

[  |  Validation  and  verification  plan 

Q  Acquisition  plan  /  Subcontracting  planning 

fl  Deployment  or  Installation  plan 1  Marketing  plan 

1  |  Quality  requirements  /  Quality  assurance  plan 

□  Risk  management  planning 
1  I  Project  glossary 

j  |  Project  communications  planning 
I  |  Project  organization  charts 
|  |  Staff  responsibilities  and  responsibility  definitions 
I  I  Necessary  facility,  equipment,  and  component  requirements 
I  I  None 

PPE7.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

I  I  There  is  a  statement  of  work  (or  a  similar  document)  stating  what  needs  to  be 
accomplished/done . 

There  is  a  work  breakdown  structure  or  a  feature/functionality  list  (or  a  similar 
document)  that  details  the  project  tasks/activities. 
i  |  The  tasks  and  activities  are  identified  as  the  project  progresses. 

I  I  None 

PPE8.  What  kinds  of  effort,  schedule  or  cost  estimation  techniques  are  used?  (Check 
all  that  apply.) 

□  Experiences  of  project  manager/management  team 
|  |  Inputs  from  project  team  members 

|~|  Expert  or  consultant  judgment 
1  I  Analogy  to  similar  projects 
[Q  Historical  data 
I  I  Automated  cost  estimation  tools 
I  I  None 

PPE9.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

I  I  No  estimation  is  needed. 

f~l  Only  one  type  of  estimation  technique  is  used. 

|  |  Two  or  more  estimation  techniques  are  used. 

I  |  Estimates  from  various  techniques  are  compared  and  analyzed  for  discrepancies. 

I  I  None 
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PPE10.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

□  Lines  of  code  (LOC)  are  used  in  estimation. 

I  |  Function  points  are  used  in  estimation. 

I  I  Number  of  functionality/features  are  used  in  estimation. 
i  |  Number  of  modules  and  deliverables  are  used  in  estimation. 

I  I  Other  advanced  metrics  used  in  estimation. 

I  I  None 


(5)  (4)  (3)  (2)  (1)  N/A 


PPE 

11 

The  project  schedule  is 
feasible. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PPE 

12 

The  funding  for  the  project  is 
adequate. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PPE 

13 

The  project  is  adequately 
staffed. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PPE 

14 

Extra  funding  for 
unprecedented  issues  is  set 
aside. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PPE 

15 

Slack  or  buffer  time  exists  in 
the  schedule  for 
unprecedented  or  extra 
activities. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PPE 

16 

Alternative  staff  to  accomplish 
critical  tasks/activities  are 
considered  and  incorporated 
in  the  project  plan. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PPE 

17 

All  relevant  stakeholders  are 
identified  before  planning 
activities. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 
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PPF 

A  certain  level  of  requirements 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

i  JL  XL/ 

18 

analysis  is  conducted  before 
planning  and  estimation. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

All  external  dependencies  are 
identified  and  incorporated  to 

PPE 

the  planning.  (Such  as 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

19 

acquisition  of  various  products 
and  services  from  outside 
vendors,  required  permissions 

□ 

□ 

□ 

□ 

□ 

□ 

PPE 

20 

The  project  plan  is  updated 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

throughout  the  project 

development. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

The  project  plan  is 

PPE 

visible/available  to  project 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

21 

team  members  and  other 
relevant  stakeholders. 

□ 

□ 

□ 

□ 

□ 

□ 

PPE 

22 

Various  automated  project 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

management  tools  are  used  in 
planning  the  project. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

PPE 

23 

The  project  team  members  are 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

consulted  in  planning  and 
estimation  efforts. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

PPE 

24 

The  managers  at  various  levels 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

have  project  planning  and 
estimation  training. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

Each  task/activities/work 

PPE 

packages  are  assigned  to 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

25 

specific  project  team  member 
or  members. 

□ 

□ 

□ 

□ 

□ 

□ 
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PPE 

26 

Critical  activities  are 

identified  and/or  critical  path 
analysis  is  conducted. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PPE 

27 

Various  standards,  guidelines 
or  checklists  are  used  in 
planning  and  estimation. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PPE 

28 

Formal  analysis  is  conducted 
for  cost,  schedule  and  effort 
estimation  such  as  PERT, 
CPM  etc. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PPE 

29 

Factors  such  as  staff  turnover 
or  loss  of  key  personnel  are 
considered  during  planning. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PPE 

30 

Realistic  estimates  guide  the 
project  planning. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PPE 

31 

Testing  is  carefully 

incorporated  to  project  plan. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PPE 

32 

Effort  estimations  are 

provided  by  those  performing 
the  tasks. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PPE 

33 

Project  risks  are  carefully 
analyzed  and  contingencies 
are  included  in  the  planning. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 
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PPE 

34 

A  suitable  project 

development  approach  and 
process  is  identified  with 
rationale  in  the  project  plan. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

PPE 

35 

All  necessary  skills  and 
expertise  needed  in  the  project 
are  identified. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 
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K.  SCOPE  MANAGEMENT  SECTION  (16  QUESTIONS  -  ABOUT  3-8 
MINUTES) 

SMI.  Check  the  statement  that  applies  to  the  project.  (Check  only  one.) 

f~|  Project  scope  never  changed. 

P  Project  scope  frequently  changed. 

I  I  Project  scope  somewhat  changed. 

SM2.  Check  the  statement  that  applies  to  the  project.  (Check  only  one.) 

I  |  Project  scope  is  ambiguous  at  first  and  it  becomes  clear  during  the  project. 

H  Project  scope  is  ambiguous  at  first  and  stays  ambiguous  due  to  various  reasons. 

|  |  Project  scope  is  defined  and  clear  at  the  beginning  of  the  project  and  it  stays  clear. 

I  I  Project  scope  is  defined  and  clear  at  the  beginning  of  the  project  and  it  become 
ambiguous  due  to  various  reasons. 

SM3.  Check  the  statement  that  applies  to  the  project.  (Check  only  one.) 

;  |  There  is  a  project  scope  document  and  it  stayed  the  same  from  the  project  start. 

P  There  is  a  project  scope  document  and  it  is  updated  when  it  is  necessary. 

I  I  There  isn’t  a  project  scope  document. 

SM4.  What  is  the  effect  of  project  scope  changes  on  the  project  schedule?  (Check 
only  one.) 

I  |  None 

I  I  On  time  without  scope  change/s 
I  I  On  time  with  scope  change/s 
i  |  Late  without  scope  change/s 
I  I  Late  with  scope  change/s 

SM5.  What  is  the  effect  of  project  scope  changes  on  the  project  budget?  (Check  only 
one.) 

[~l  None 

n  Within  budget  without  scope  change/s 
I  |  Within  budget  with  scope  change/s 
I  |  Cost  overrun  without  scope  change/s 
I  I  Cost  overrun  with  scope  change/s 

SM6.  What  is  the  effect  of  project  scope  changes  on  the  functionality  of  the 
deliverables?  (Check  only  one.) 

P  None 

P  Full  functionality  without  scope  change/s 
I  ~|  Full  functionality  with  scope  change/s 
O  Less  than  planned  functionality  without  scope  change/s 
I  |  Less  than  planned  functionality  with  scope  change/s 
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SM7.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

I  |  Project  scope  changes  are  handled  only  by  the  management. 

I  |  Project  scope  changes  have  to  follow  a  formal  defined  process. 

I  I  Project  scope  changes  follow  a  decision-making  process  that  includes  management, 
stakeholders,  and  team  members. 

I  I  Project  scope  changes  handled  informally  by  the  management. 

SM8.  Which  of  the  following  statement/s  is/are  included  in  the  project  scope 
document,  if  there  is  one.  (Check  all  that  apply.) 

I  |  The  problem  statement 

□  The  work  to  be  done  or  work  breakdown  structure 
U  The  constraints 

1  I  The  resources 

CH  Preliminary  or  detailed  schedule  and  cost  analysis 
1  I  The  project  deliverables 

i  |  Clear  definition  of  performance  to  meet  contractual  and  legal  obligations 

□  Glossary 

I  |  Not  Available 

SM9.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

I  |  The  project  scope  is  defined  after  stakeholders  are  identified. 

I  |  There  is  at  least  one  project  scope  identification/definition  meeting  at  the  beginning  of 
the  project. 

I  I  There  is  a  project  scope  change  board. 

SM10.  Who  are  included  while  defining  and  updating  the  project  scope?  (Check  all 
that  apply.) 

I  |  Project  management  team 
j  |  Project  manager 
I  I  All  stakeholders 
I  I  Some  stakeholders 
|  |  Project  team  members 

1  |  Subcontractor  representatives  if  there  is  subcontracting 
I  I  None 
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(5) 

(4) 

(3) 

(2) 

(1) 

N/A 

SM11 

Before  defining  the  project 
scope,  there  is  a  rigorous 
information  gathering  activity 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

about  the  problem  that  is  to  be 
solved,  the  resources,  the 
constraints,  the  deliverables 

□ 

□ 

□ 

□ 

□ 

□ 

SM12 

Project  scope  is  not  clearly 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

defined  due  to  various  reasons. 

□ 

□ 

□ 

□ 

□ 

□ 

The  project  has  a  documented 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

SM13 

project  scope  definition  and  a 
formal  scope  change  process. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

SM14 

Project  scope  is  always  visible 
and  clear  to  stakeholders, 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

project  team  members,  and 
management. 

□ 

□ 

□ 

□ 

□ 

□ 

Project  scope  changes  have  to 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

SMI  5 

go  through  an  extensive 
decision-making  process. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

The  project  scope  document  is 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

SM16 

reviewed  and  approved  by  all 
stakeholders. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 
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L.  RISK  CONTROL  SECTION  (17  QUESTIONS  -  ABOUT  3-8  MINUTES) 

RC1.  What  is  the  overall  risk  level  of  the  project?  (Check  only  one.) 

□  High 
EH  Medium 
I  I  Low 

□  None 

RC2.  What  is  the  effect  of  risks  on  the  project  budget?  (Check  only  one.) 

1  I  High  cost  overrun 
I  I  Medium  cost  overrun 
I  I  Low  cost  overrun 

□  None 

RC3.  What  is  the  effect  of  risks  on  the  project  schedule?  (Check  only  one.) 

EH  The  project  delivery  is  on  time. 

CD  The  project  delivery  is  slightly  late. 

I  I  The  project  delivery  is  significantly  late. 

RC4.  What  is  the  effect  of  risks  on  the  project  functionality?  (Check  only  one.) 

□  High 
EH  Medium 
EH  Low 

□  None 

RC5.  What  is  the  level  of  funding  and  resources  set  aside  for  risk  management?  (Check  only  one.) 

I  I  More  than  enough 
EH  Enough 
EH  Hardly  enough 
EH  No  funding  and  resources 

RC6.  Check  the  statement/s  that  applies  to  the  project.  (Check  only  one.) 

EH  Adequate  slack  time  is  planned  in  the  schedule  for  consequences  due  to  risks. 

EH  There  is  not  any  slack  time  planned  for  consequences  due  to  risks. 

EH  Not  enough  slack  time  is  planned  in  the  schedule  for  consequences  due  to  risks. 

RC7.  Check  the  statement  that  applies  to  the  project.  (Check  only  one.) 

EH  Risks  are  handled  when  they  occur. 

EH  Risks  are  addressed  before  they  occur. 

EH  Both 

RC8.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

EH  Informal  project  risk  management  procedures  are  in  place. 

EH  Project  risk  management  is  based  on  formal  procedures. 

EH  There  is  not  any  project  risk  management  and  planning. 

RC9.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

EH  Risks  are  generally  avoided.  (Risk  Avoidance) 

EH  Risks  are  transferred  to  third  parties  for  example  contracting  risky  development  items  to  consultants  or 
experts.  (Risk  Transfer) 

EH  Risks  are  managed  as  they  occur. 

EH  Risk  mitigation  (actions  reducing  the  severity/impact  of  a  risk)  is  the  most  used  option  in  risk 
management  of  the  project.  (Risk  Mitigation) 

EH  None 
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RC10.  Check  the  statement  that  applies  to  the  project.  (Check  only  one.) 

I  I  Experts  are  consulted  in  the  risk  management  of  the  project. 

I  I  Project  management  handles  all  the  risks. 

I  I  Project  team  members  and  stakeholders  are  involved  in  the  risk  management. 


(5) 

(4) 

(3) 

(2) 

(1) 

N/A 

RCll 

For  each  identified  risk  item, 
there  is  an  information 
gathering  activity. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

RC12 

Contingencies  and  alternative 
solutions  are  planned  for  the 
critical  tasks  and  portions  of 
the  development  exposed  to 
high  risks. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

RC13 

Top  risk  items  list  is  closely 
monitored  and  periodically 
updated. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

RC14 

Risk  monitoring  is  an 
important  activity  in  the 
project. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

RC15 

Risk  avoidance  is  primary 
method  of  risk  control 
activities. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

RC16 

There  are  regular  project  risk 
monitoring  meetings  or 
project  risk  monitoring  is 
handled  through  project  status 
meetings  etc. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

RC17 

There  is  a  risk  management 
plan  and  course  of  action  for 
each  high-risk  items. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 
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M.  STAFFING/HIRING  SECTION(29  QUESTIONS  -  ABOUT  7-13 
MINUTES) 

51.  Which  of  the  followings  are  clearly  identified,  documented  and  communicated?  (Check  all  that 
apply.) 

□  Project  Roles 

I  I  Project  Positions 

I  I  Necessary  Qualifications  for  the  project 

□  None 

52.  Which  of  the  documents  or  similar  documents  exist  for  the  project?  (Check  all  that  apply.) 

I  I  Project  staffing  management  plan 

I  I  Project  responsibility/accountability/interfaces/assignment  matrix 
CD  Project  work  breakdown  structure 

□  None 

53.  What  is  the  experienced-to-inexperienced  project  team  member  ratio?  (experienced: 
inexperienced)  (Check  only  one.) 

I  |  Smaller  than  1:2 

□  1:2 
□  1:1 
□  2:1 

I  I  Greater  than  2:1 

54.  Which  of  the  followings  for  team  members  are  clearly  identified,  documented  and 
communicated?  (Check  all  that  apply.) 

□  Responsibility 

□  Job  Interfaces 

□  Reporting  Structure 

□  None 


(5) 

(4) 

(3) 

(2) 

(1) 

N/A 

S5 

The  work  breakdown 

structure  (WBS)  or  similar 
document  is  completed  before 
hiring/staffing. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

S6 

The  analysis  of  the  required 
work  and  resources  is 
conducted  rigorously. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

S7 

Significant  project  risks  are 
identified  before  the 

hiring/staffing  the  team. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 
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S8 

There  is  adequate  funding  and 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

resources  for  hiring/staffing. 

□ 

□ 

□ 

□ 

□ 

□ 

There  are  adequate  work  force 
and  experts  with  the  necessary 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

S9 

skills  and  expertise  available 
for  hiring  and/or  staffing  on 
this  project. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

Expertise  on  human  resources 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

S10 

is  acquired  for  staffing  and 
hiring  activities. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

Project  open  positions  are 
made  attractive  to  qualified 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

Sll 

candidates  through  incentives 
etc.  The  position  is  made 
desirable. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

S12 

The  skills  and  expertise 
needed  for  the  project  success 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

are  acquired  with  the  timely 
recruitment  of  team  members. 

□ 

□ 

□ 

□ 

□ 

□ 

S13 

The  necessary  interpersonal 
skills  for  the  roles  are 
identified  and  the  project  team 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

members  are  recruited  also 
based  on  their  interpersonal 
skills. 

□ 

□ 

□ 

□ 

□ 

□ 

S14 

The  ambitions  and  goals  of  the 
project  team  members  are 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

aligned  with  the  project 
mission  and  goals. 

□ 

□ 

□ 

□ 

□ 

□ 

The  project  team  members 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

S15 

have  the  necessary  educational 
background. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 
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S16 

The  project  team  members 
have  similar  project  work 
experience. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

S17 

The  productivity  of  the  project 
team  members  are  within  the 
expectations. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

S18 

Project  team  members  are 
familiar  and  comfortable  with 
the  organizational  culture. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

S19 

Project  team  members  have 
difficulties  with  the 

organizational  procedures. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

S20 

Project  team  members  are 
happy  with  their  roles, 
positions  and  career 
advancement  opportunities  in 
the  project  organization. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

S21 

Project  team  members  stay 
with  the  project  according  to 
the  project  staffing 
management  plan.  Turn-over 
rate  is  at  minimum. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

S22 

Resignations  are  at  minimum. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

S23 

Project  team  members  acquire 
the  necessary  skills  and 
expertise  needed  for  the 
project  through  training  and 
coaching. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 
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There  are  alternative  team 

S24 

members  with  the  necessary 
skills  and  knowledge  to  take 
over  some  other  team 
member’s  work  for  critical 
tasks  in  case  of  team  member 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

S25 

Project  positions  are  filled 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

with  qualified  individuals. 

□ 

□ 

□ 

□ 

□ 

□ 

S26 

Work  and  task  assignments 
are  fair  and  based  on 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

qualifications  of  the  project 
team  members. 

□ 

□ 

□ 

□ 

□ 

□ 

S27 

Removing  of  project  team 
members  for  unsatisfactory 
work  performance  and/or 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

other  reasons  are  conducted 
fairly  and  according  to  the 
organizational  procedures. 

□ 

□ 

□ 

□ 

□ 

□ 

S28 

Orientation  or  transition 
activities  for  the  new  team 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

members  are  conducted 
properly. 

□ 

□ 

□ 

□ 

□ 

□ 

When  necessary,  consultants 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

S29 

and  contractors  are  used 

Agree 

□ 

□ 

□ 

Disagree 

Applicable 

effectively. 

□ 

□ 

□ 
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N.  CONFIGURATION  MANAGEMENT  SECTION  (13  QUESTIONS  - 
ABOUT  3-7  MINUTES) 

*  In  some  organizations  configuration  management  is  referred  as  version  control. 

CM1.  Check  the  statement  that  applies  to  the  project.  (Check  only  one.) 

I  |  Configuration  management  is  conducted  informally. 

I  I  Configuration  management  is  a  fonnal  and  documented  activity  and  it  has  well- 
defined  procedures. 

CM2.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

I H  There  is  a  configuration  management  document. 

|  |  There  is  a  configuration  or  change  control  board,  committee  or  team. 

I  I  There  is  a  configuration  items  list. 

I  I  None 

CM3.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

I  |  Baselines  and  configuration  items  are  identified  at  the  beginning  of  the  project  and 
updated  as  necessary. 

j  ~|  The  owner  or  responsible  staff  is  identified  for  each  configuration  item. 

I  I  Every  configuration  item  has  a  unique  identifier. 

I  |  Important  characteristics  for  each  configuration  item  are  identified  such  as  author, 
type,  date,  version  number  etc. 

Pi  None 

CM4.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

I  |  The  configuration  management  procedures  includes  a  detailed  change  and  change 
request  protocols. 

I  |  The  configuration  management  system  has  various  levels  of  control  (such  as  only 
author  may  release  the  item,  restricted  write  access  etc). 

I  |  There  is  not  a  configuration  management  system  and  configuration  management  is 
only  the  responsibility  of  project  team  members  or  developers. 

I  I  None 

CM5.  Check  the  statement  that  applies  to  the  project.  (Check  only  one.) 

j~~l  The  change  requests  have  to  go  through  the  change  control  board  or  responsible  staff. 

I  I  The  change  requests  are  only  handled  by  the  developer  or  the  owner  of  the 
configuration  item. 
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(5) 

(4) 

(3) 

(2) 

(1) 

N/A 

The  project  suffers  from 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

CM6 

configuration/version 
management  problems. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

An  automated  configuration 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

CM7 

management  system  is  used 
and  adequate  for  the  project. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

The  configuration 
management  procedures  are 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

CM8 

strictly  followed.  Project  team 
members  do  not  try  to  bypass 
them. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

The  integrity,  security  and 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

CM9 

privacy  of  configuration  items 
are  satisfactory. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

The  changes  and  change 

CM 

requests  are  controlled,  and 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

10 

documented  in  such  a  way  that 
it  enables  audit. 

□ 

□ 

□ 

□ 

□ 

□ 

CM 

11 

Every  change  request  is 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

controlled  and  extensively 
reviewed. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

CM 

12 

Records  of  configuration 
management  activities, 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

changes  to  baselines,  work 
products,  and  change  requests 
are  well-maintained. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

CM 

13 

There  is  an  established  and 
reliable  configuration 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

management  system  including 
automated  tools,  databases, 
protocols  etc. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 
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O.  RISK  ASSESSMENT  SECTION  (20  QUESTIONS  -  ABOUT  5-10 
MINUTES) 

RA1.  Which  of  the  following  does  best  characterize  the  risk  assessment  activities  in 
the  project?  (Check  only  one.) 

H  Formal 
I  |  Informal 

□  Semiformal 

I  I  Not  available 

RA2.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

H  Risks  are  assessed  as  they  are  identified  during  the  project. 

□  Risks  are  assessed  early  and  incorporated  into  a  risk  management  document. 

I  |  The  risk  management  document  is  periodically  updated. 

|  |  There  is  staff  specifically  assigned  to  risk  assessment  activities. 

I  I  Lessons  learned  are  visited  prior  to  risk  assessment  activities. 

I  I  None 

RA3.  In  which  of  the  following  categories  the  risks  are  assessed  and  documented? 
(Check  all  that  apply.) 

f~l  People 

□  Schedule 

□  Budget  and  Funding 
I  I  Technology 

□  Requirements 
I  |  Subcontractor 
I  I  None 

RA4.  There  are  common  objective  criteria  to  assess  risks.  (Check  only  one.) 

□  Yes 

□  No 

□  Partially 

I  |  Not  Available 

RA5.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

j  I  There  is  a  project  risk  management  plan. 

I  I  The  project  risk  management  plan  includes  objective  criteria  for  risk  identification, 
analysis  and  prioritization. 

I  |  Project  risk  document  is  updated  frequently  along  the  project. 

I  I  None 

RA6.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

I  |  Experts  or  consultants  are  used  for  risk  assessment. 

□  Experienced  project  staff  is  used  for  risk  assessment. 

□  Project  manager  conducted  the  risk  assessment. 

I  |  There  is  not  any  risk  assessment  activity. 
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RA7.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

□  Risks  are  identified. 

I  |  Risks  are  analyzed. 

□  Risks  are  categorized. 

I  |  Risks  are  prioritized. 

I  I  None 

RA8.  Check  the  statement  that  applies  to  the  project.  (Check  only  one.) 

Q  Risk  assessment  is  based  on  qualitative  methods. 

I  |  Risk  assessment  is  based  on  quantitative  methods. 

I  I  Risk  assessment  is  based  on  the  judgment  of  the  management. 

I  |  Risk  assessment  is  based  on  both  qualitative  and  quantitative  methods. 

I  I  There  is  no  need  for  any  risk  assessment  activity. 


(5) 

(4) 

(3) 

(2) 

(1) 

N/A 

RA9 

The  projects  risks  are 
documented  early  with  details 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

related  to  their  impact  on  the 
project. 

□ 

□ 

□ 

□ 

□ 

□ 

Risk  assessment  has  a  clear 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

RA10 

impact  on  project  planning 
and  decisions. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

RA11 

Sufficient  reserve  resources 
and  funding  are  planned  and 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

set  aside  for  risk  assessment 
activities. 

□ 

□ 

□ 

□ 

□ 

□ 

RA12 

Top  risk  items  list  or  a  similar 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

list  is  maintained. 

□ 

□ 

□ 

□ 

□ 

□ 

RA13 

Risks  are  assessed  with  the 
broad  inclusion  of 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

stakeholders  and  project  team 
members. 

□ 

□ 

□ 

□ 

□ 

□ 
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RA14 

Project  environment  facilitates 
and  encourages  open  and  free 
discussions  on  project  risks. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

RA15 

Risks  are  identified  using  risk 
identification  tools  such  as 
checklists,  databases,  risk 
taxonomy,  decision-driver 
analysis  etc. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

RA16 

Risks  are  analyzed  based  on 
their  probability  of  occurrence 
and  impact  on  the  project. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

RA17 

Risks  are  prioritized  based  on 
their  probability  of  occurrence 
and  impact  on  the  project. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

RA18 

Risk  assessment  information  is 
always  visible  and  they  are 
shared  with  stakeholders  and 
project  team  members. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

RA19 

Any  stakeholder  or  project 
team  member  may  report  a 
risk  at  any  time  and  there  is  a 
mechanism  allowing  such 
reports. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

RA20.  (Answer  only  if  a  portion  of  the  system  is  subcontracted.)  Check  the 
statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

j  |  Subcontractor/s  is/are  free  in  their  risk  management  decision  and  activities. 

I  I  Subcontractor/s  is/are  contractually  responsible  to  have  formal  risk  assessment 
procedures. 

[ H  Subcontractor/s  is/are  contractually  responsible  to  deliver  risk  assessment  reports. 

I  I  Subcontractor/s  has/have  a  representative  for  project  risk  management  meetings. 
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P.  QUALITY  ENGINEERING  SECTION  (20  QUESTIONS  -  ABOUT  4-10 
MINUTES) 

QE1.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

[~|  There  is  a  quality  policy. 

O  Quality  is  not  a  high  priority  in  this  project  due  to  various  reasons. 

I  I  There  is  a  quality  planning  activity. 

QE2.  Check  the  statement/s  that  applies  to  the  project.  (Check  all  that  apply.) 

I  |  Quality  expectations  of  various  stakeholders  are  identified  and  documented. 

I  I  The  quality  standards  and  guidelines  related  to  the  project  are  identified.  (Such  as 
aviation  standards  etc.) 

I  I  Objective  quality  criteria  for  the  project  and  its  deliverables  are  identified. 

I  I  None 

QE3.  Which  of  the  following  quality  attribute/s  are  considered  achieved  in  the 
project?  (Check  all  that  apply.) 

j  |  Maintainability 
□  Safety 
I  I  Security 
I  |  Reliability 
H  Usability 
fl  Other 
I  I  None 

QE4.  What  is  the  amount  of  testing  conducted  during  the  project  development? 
(Check  only  one.) 

|  J  Extensive 
f~|  Fair 
n  Some 
I  I  None 


(5) 

(4) 

(3) 

(2) 

(1) 

N/A 

QE5 

Quality  is  considered  a  high 
priority  in  this  project. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 

QE6 

There  is  support  for  and 
commitment  to  quality  from 
executive  management. 

Completely 

Agree 

□ 

Agree 

□ 

Neutral 

□ 

Disagree 

□ 

Completely 

Disagree 

□ 

Not 

Applicable 

□ 
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QE7 

High  quality  is  planned  from 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

the  start  in  this  project. 

□ 

□ 

□ 

□ 

□ 

□ 

QE8 

Various  quality  metrics  are 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

identified. 

□ 

□ 

□ 

□ 

□ 

□ 

QE9 

Quality  assurance  procedures 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

are  adequate. 

□ 

□ 

□ 

□ 

□ 

□ 

QE10 

Quality  assurance  procedures 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

are  documented. 

□ 

□ 

□ 

□ 

□ 

□ 

Adequate  amount  of  resources 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

QE11 

are  set  aside  for  quality 
engineering  activities. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

The  requirements  are  defined 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

QE12 

with  the  guidance  of  quality 
expectations. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

The  project  team  culture 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

QE13 

encourages  commitment  to 
high  quality. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

QE14 

Project  team  members  are 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

trained  in  quality  assurance. 

□ 

□ 

□ 

□ 

□ 

□ 
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There  are  quality  thresholds 
and  expectations  for  various 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

QE15 

work  products  such  as  system 
architecture,  requirements 
definitions,  designs,  testing  etc. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

QE16 

Quality  considerations  are 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

limited  to  testing. 

□ 

□ 

□ 

□ 

□ 

□ 

QE17 

High  testing  coverage  for  the 

Completely 

Agree 

Agree 

Neutral 

Disagree 

Completely 

Disagree 

Not 

Applicable 

product  is  achieved. 

□ 

□ 

□ 

□ 

□ 

□ 

There  are  adequate  tools, 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

QE18 

equipment  and  resources  for 
testing. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

There  are  specifically  assigned 

Completely 

Agree 

Neutral 

Disagree 

Completely 

Not 

QE19 

team  members  for  quality 
related  issues. 

Agree 

□ 

□ 

□ 

□ 

Disagree 

□ 

Applicable 

□ 

QE20.  Which  of  the  following  activity  or  activities  are  conducted  during  the  project 
development?  (Check  all  that  apply.) 

H  Design  reviews 
[~~1  Code  reviews/inspections 
I  |  Performance  testing 
|  |  Independent  verification  and  validation 
1  I  Quality  assurance  activities 
1  |  Requirements  tracing 
I  |  Various  types  of  testing 
I  |  Defect  identification  and  prevention 
I  I  Simulations  and/or  prototyping 
I  I  None 
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APPENDIX  G:  SOFTWARE  PROJECT  MANAGEMENT 
EVALUATION  MODEL  SHEET  BY  SECTION 


A.  COMMUNICATION  SECTION 


Choices 

Question  Number 

A 

B 

C 

D 

E 

F 

G 

Cl 

2 

2 

2 

2 

2 

0 

C2 

1 

1 

1 

1 

1 

1 

C3 

1 

1 

1 

1 

1 

1 

0 

C4 

1 

1 

1 

1 

1 

1 

0 

Choices 

Question  Number 

5 

4 

3 

2 

1 

N/A 

C5 

2 

1 

0 

-1 

-2 

0 

C6 

2 

1 

0 

-1 

-2 

0 

C7 

-2 

-1 

0 

1 

2 

0 

C8 

2 

1 

0 

-1 

-2 

0 

C9 

2 

1 

0 

-1 

-2 

0 

CIO 

2 

1 

0 

-1 

-2 

0 

Cll 

2 

1 

0 

-1 

-2 

0 

C12 

2 

1 

0 

-1 

-2 

0 

C13 

2 

1 

0 

-1 

-2 

0 

C14 

2 

1 

0 

-1 

-2 

0 

C15 

2 

1 

0 

-1 

-2 

0 

C16 

2 

1 

0 

-1 

-2 

0 

C17 

-2 

-1 

0 

1 

2 

0 

C18 

2 

1 

0 

-1 

-2 

0 

C19 

2 

1 

0 

-1 

-2 

0 

C20 

2 

1 

0 

-1 

-2 

0 

C21 

2 

1 

0 

-1 

-2 

0 

C22 

2 

1 

0 

-1 

-2 

0 

C23 

2 

1 

0 

-1 

-2 

0 
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LEADERSHIP  SECTION 


Choices 

Question  Number 

5 

4 

3 

2 

1 

N/A 

LI 

-2 

-1 

0 

1 

2 

0 

L2 

2 

1 

0 

-1 

-2 

0 

L3 

2 

1 

0 

-1 

-2 

0 

L4 

2 

1 

0 

-1 

-2 

0 

L5 

2 

1 

0 

-1 

-2 

0 

L6 

2 

1 

0 

-1 

-2 

0 

L7 

2 

1 

0 

-1 

-2 

0 

L8 

2 

1 

0 

-1 

-2 

0 

L9 

2 

1 

0 

-1 

-2 

0 

L10 

2 

1 

0 

-1 

-2 

0 

Lll 

-2 

-1 

0 

1 

2 

0 

L12 

-2 

-1 

0 

1 

2 

0 

L13 

2 

1 

0 

-1 

-2 

0 

L14 

2 

1 

0 

-1 

-2 

0 

L15 

2 

1 

0 

-1 

-2 

0 

L16 

2 

1 

0 

-1 

-2 

0 

Question  Number 

Choices 

A 

B 

c 

D 

L17* 

2 

2 

1 

-2 

L18* 

-2 

1 

2 

2 

*Either  provide  responses  to  L17  or  L18 


D. 


ORGANIZATIONAL  COMMITMENT  SECTION 


Choices 

Question  Number 

5 

4 

3 

2 

1 

N/A 

OC1 

2 

1 

0 

-1 

-2 

0 

OC2 

2 

1 

0 

-1 

-2 

0 

OC3 

2 

1 

0 

-1 

-2 

0 

OC4 

2 

1 

0 

-1 

-2 

0 

OC5 

2 

1 

0 

-1 

-2 

0 

OC6 

2 

1 

0 

-1 

-2 

0 

OC7 

2 

1 

0 

-1 

-2 

0 

OC8 

2 

1 

0 

-1 

-2 

0 

OC9 

2 

1 

0 

-1 

-2 

0 

OCIO 

2 

1 

0 

-1 

-2 

0 

OC11 

2 

1 

0 

-1 

-2 

0 

OC12 

2 

1 

0 

-1 

-2 

0 

OC13 

2 

1 

0 

-1 

-2 

0 

OC14 

2 

1 

0 

-1 

-2 

0 

OC15 

2 

1 

0 

-1 

-2 

0 

OC16 

2 

1 

0 

-1 

-2 

0 

OC17 

2 

1 

0 

-1 

-2 

0 

OC18 

-2 

1 

0 

-1 

-2 

0 

OC19 

-2 

1 

0 

-1 

-2 

0 

OC20 

2 

1 

0 

-1 

-2 

0 

OC21 

2 

1 

0 

-1 

-2 

0 

OC22 

2 

1 

0 

-1 

-2 

0 

OC23 

2 

1 

0 

-1 

-2 

0 

OC24 

2 

1 

0 

-1 

-2 

0 

Question  Number 

Choices 

A 

B 

C 

D 

E 

F 

OC25 

2 

2 

2 

0 

OC26 

2 

2 

2 

-2 

2 

0 
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E. 


PROJECT  MANAGER  SECTION 


Question  Number 

Choices 

A 

B 

C 

D 

E 

F 

PM1 

0 

-2 

-4 

-6 

PM2 

0 

1 

2 

3 

4 

PM3 

1 

1 

1 

1 

1 

0 

PM4 

1 

1 

1 

1 

1 

0 

Choices 

Question  Number 

5 

4 

3 

2 

1 

N/A 

PM5 

2 

1 

0 

-1 

-2 

0 

PM6 

2 

1 

0 

-1 

-2 

0 

PM7 

2 

1 

0 

-1 

-2 

0 

PM8 

2 

1 

0 

-1 

-2 

0 

PM9 

2 

1 

0 

-1 

-2 

0 

PM10 

2 

1 

0 

-1 

-2 

0 

PM11 

2 

1 

0 

-1 

-2 

0 

PM12 

2 

1 

0 

-1 

-2 

0 

PM13 

2 

1 

0 

-1 

-2 

0 

PM14 

2 

1 

0 

-1 

-2 

0 

PM15 

2 

1 

0 

-1 

-2 

0 

PM16 

2 

1 

0 

-1 

-2 

0 

PM17 

2 

1 

0 

-1 

-2 

0 

PM18 

2 

1 

0 

-1 

-2 

0 

PM19 

2 

1 

0 

-1 

-2 

0 

PM20 

2 

1 

0 

-1 

-2 

0 

PM21 

2 

1 

0 

-1 

-2 

0 

PM22 

2 

1 

0 

-1 

-2 

0 

PM23 

2 

1 

0 

-1 

-2 

0 

PM24 

2 

1 

0 

-1 

-2 

0 

PM25 

2 

1 

0 

-1 

-2 

0 

PM26 

2 

1 

0 

-1 

-2 

0 

PM27 

2 

1 

0 

-1 

-2 

0 
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F.  REQUIREMENTS  MANAGEMENT  SECTION 


Question  Number 

Choices 

A 

B 

C 

D 

E 

F 

RM1 

2 

2 

2 

2 

0 

RM2 

-2 

2 

2 

-2 

0 

RM3 

1 

1 

1 

1 

1 

0 

RM4 

2 

2 

2 

2 

2 

0 

Choices 

Question  Number 

5 

4 

3 

2 

1 

N/A 

RM5 

2 

1 

0 

-1 

-2 

0 

RM6 

2 

1 

0 

-1 

-2 

0 

RM7 

2 

1 

0 

-1 

-2 

0 

RM8 

2 

1 

0 

-1 

-2 

0 

RM9 

2 

1 

0 

-1 

-2 

0 

RM10 

2 

1 

0 

-1 

-2 

0 

RM11 

2 

1 

0 

-1 

-2 

0 

RM12 

2 

1 

0 

-1 

-2 

0 

RM13 

2 

1 

0 

-1 

-2 

0 

RM14 

2 

1 

0 

-1 

-2 

0 

RM15 

2 

1 

0 

-1 

-2 

0 

RM16 

2 

1 

0 

-1 

-2 

0 

RM17 

2 

1 

0 

-1 

-2 

0 

RM18 

2 

1 

0 

-1 

-2 

0 

RM19 

2 

1 

0 

-1 

-2 

0 

RM20 

2 

1 

0 

-1 

-2 

0 

RM21 

2 

1 

0 

-1 

-2 

0 

RM22 

2 

1 

0 

-1 

-2 

0 

RM23 

2 

1 

0 

-1 

-2 

0 

RM24 

2 

1 

0 

-1 

-2 

0 

RM25 

2 

1 

0 

-1 

-2 

0 

RM26 

-2 

1 

0 

-1 

-2 

0 

RM27 

2 

1 

0 

-1 

-2 

0 
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G. 


STAKEHOLDER  INVOLVEMENT  SECTION 


Choices 

Question  Number 

5 

4 

3 

2 

1 

N/A 

SI1 

2 

1 

0 

-1 

-2 

0 

SI2 

2 

1 

0 

-1 

-2 

0 

SI3 

2 

1 

0 

-1 

-2 

0 

SI4 

2 

1 

0 

-1 

-2 

0 

SI5 

2 

1 

0 

-1 

-2 

0 

SI6 

2 

1 

0 

-1 

-2 

0 

SI7 

2 

1 

0 

-1 

-2 

0 

SI8 

2 

1 

0 

-1 

-2 

0 

SI9 

2 

1 

0 

-1 

-2 

0 

Question  Number 

Choices 

A 

B 

C 

D 

E 

F 

G 

SI10 

2 

2 

2 

2 

2 

2 

0 

Sill  and  SI12  are  to  be  answered  if  the  project  is  developed 
for  the  market  without  a  specific  contract. 


Question  Number 

5 

4 

3 

2 

1 

N/A 

Sill 

2 

1 

0 

-1 

-2 

0 

SI12 

2 

1 

0 

-1 

-2 

0 

SI13-SI18  are  to  be  answered  if  the  project  is  developed  under 
a  contract  with  a  specific  customer. 


Choices 

Question  Number 

5 

4 

3 

2 

1 

N/A 

SI13 

-2 

-1 

0 

1 

2 

0 

SI14 

2 

1 

0 

-1 

-2 

0 

SI15 

2 

1 

0 

-1 

-2 

0 

SI16 

2 

1 

0 

-1 

-2 

0 

SI17 

2 

1 

0 

-1 

-2 

0 

Question  Number 

Choices 

A 

B 

SI18 

2 

-2 
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H. 


PROJECT  MONITORING  AND  CONTROL  SECTION 


Question  Number 

Choices 

A 

B 

C 

D 

E 

F 

G 

PMC1 

2 

-2 

PMC2 

1 

1 

1 

1 

1 

1 

0 

PMC3 

2 

-2 

PMC4 

2 

2 

2 

0 

PMC  5 

2 

2 

2 

2 

2 

0 

Choices 

Question  Number 

5 

4 

3 

2 

1 

N/A 

PMC  6 

2 

1 

0 

-1 

-2 

0 

PMC  7 

-2 

1 

0 

-1 

-2 

0 

PMC  8 

2 

1 

0 

-1 

-2 

0 

PMC  9 

2 

1 

0 

-1 

-2 

0 

PMC10 

2 

1 

0 

-1 

-2 

0 

PMC11 

2 

1 

0 

-1 

-2 

0 

PMC12 

2 

1 

0 

-1 

-2 

0 

PMC13 

2 

1 

0 

-1 

-2 

0 

PMC14 

2 

1 

0 

-1 

-2 

0 

PMC15 

2 

1 

0 

-1 

-2 

0 

PMC16 

2 

1 

0 

-1 

-2 

0 

PMC17 

2 

1 

0 

-1 

-2 

0 

PMC18 

2 

1 

0 

-1 

-2 

0 

PMC19 

2 

1 

0 

-1 

-2 

0 
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I.  PROJECT  PLANNING  AND  ESTIMATION  SECTION 


Question 

Number 

Choices 

A 

B 

c 

D 

E 

F 

G 

H 

I 

J 

K 

L 

M 

N 

O 

P 

Q 

R 

PPE1 

2 

-2 

2 

-4 

PPE2 

-2 

2 

PPE3 

-2 

2 

PPE4 

2 

-2 

PPE5 

1 

1 

1 

1 

1 

1 

0 

PPE6 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

l 

0 

PPE7 

2 

2 

-4 

PPE8 

1 

1 

1 

1 

1 

1 

0 

PPE9 

-4 

0 

2 

4 

0 

PPE10 

1 

1 

1 

1 

1 

0 

Choices 

Question  Number 

5 

4 

3 

2 

1 

N/A 

PPE11 

2 

1 

0 

-1 

-2 

0 

PPE12 

2 

1 

0 

-1 

-2 

0 

PPE13 

2 

1 

0 

-1 

-2 

0 

PPE14 

2 

1 

0 

-1 

-2 

0 

PPE15 

2 

1 

0 

-1 

-2 

0 

PPE16 

2 

1 

0 

-1 

-2 

0 

PPE17 

2 

1 

0 

-1 

-2 

0 

PPE18 

2 

1 

0 

-1 

-2 

0 

PPE19 

2 

1 

0 

-1 

-2 

0 

PPE20 

2 

1 

0 

-1 

-2 

0 

PPE21 

2 

1 

0 

-1 

-2 

0 

PPE22 

2 

1 

0 

-1 

-2 

0 

PPE23 

2 

1 

0 

-1 

-2 

0 

PPE24 

2 

1 

0 

-1 

-2 

0 

PPE25 

2 

1 

0 

-1 

-2 

0 

PPE26 

2 

1 

0 

-1 

-2 

0 

PPE27 

2 

1 

0 

-1 

-2 

0 

PPE28 

2 

1 

0 

-1 

-2 

0 

PPE29 

2 

1 

0 

-1 

-2 

0 

PPE30 

2 

1 

0 

-1 

-2 

0 

PPE31 

2 

1 

0 

-1 

-2 

0 

PPE32 

2 

1 

0 

-1 

-2 

0 

PPE33 

2 

1 

0 

-1 

-2 

0 

PPE34 

2 

1 

0 

-1 

-2 

0 

PPE35 

2 

1 

0 

-1 

-2 

0 
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J.  SCOPE  MANAGEMENT  SECTION 


Question  Number 

Choices 

A 

B 

C 

D 

E 

F 

G 

H 

I 

SMI 

2 

-2 

0 

SM2 

-2 

-4 

2 

-2 

SM3 

0 

2 

-2 

SM4 

Not  Includ 

led  in  the  Model 

SM5 

Not  Included  in  the  Model 

SM6 

Not  Included  in  the  Model 

SM7 

-2 

2 

4 

-4 

SM8 

1 

1 

1 

1 

1 

1 

1 

1 

0 

SM9 

2 

2 

2 

SM10 

1 

1 

2 

1 

1 

1 

0 

Choices 

Question  Number 

5 

4 

3 

2 

1 

N/A 

SM11 

2 

1 

0 

-1 

-2 

0 

SM12 

-2 

-1 

0 

1 

2 

0 

SM13 

2 

1 

0 

-1 

-2 

0 

SM14 

2 

1 

0 

-1 

-2 

0 

SM15 

2 

1 

0 

-1 

-2 

0 

SM16 

2 

1 

0 

-1 

-2 

0 
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K. 


RISK  CONTROL  SECTION 


Question  Number 

Choices 

A 

B 

C 

D 

E 

RC1 

Not  Included  in  the  Model 

RC2 

Not  Included  in  the  Model 

RC3 

Not  Included  in  the  Model 

RC4 

Not  Included  in  the  Model 

RC5 

2 

1 

-1 

-2 

RC6 

2 

-2 

-1 

RC7 

-1 

1 

0 

RC8 

0 

2 

-2 

RC9 

1 

1 

-2 

1 

0 

RC10 

2 

-2 

2 

Choices 

Question  Number 

5 

4 

3 

2 

1 

N/A 

RC11 

2 

1 

0 

-1 

-2 

0 

RC12 

2 

1 

0 

-1 

-2 

0 

RC13 

2 

1 

0 

-1 

-2 

0 

RC14 

2 

1 

0 

-1 

-2 

0 

RC15 

2 

1 

0 

-1 

-2 

0 

RC16 

2 

1 

0 

-1 

-2 

0 
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M. 


CONFIGURATION  MANAGEMENT  SECTION 


Question  Number 

Choices 

A 

B 

C 

D 

E 

CM1 

-2 

2 

CM2 

2 

2 

2 

0 

CM3 

2 

2 

2 

2 

0 

CM4 

2 

2 

-2 

0 

CM5 

2 

-2 

Choices 

Question  Number 

5 

4 

3 

2 

1 

N/A 

CM6 

-2 

-1 

0 

1 

2 

0 

CM7 

2 

1 

0 

-1 

-2 

0 

CM8 

2 

1 

0 

-1 

-2 

0 

CM9 

2 

1 

0 

-1 

-2 

0 

CM10 

2 

1 

0 

-1 

-2 

0 

CM11 

2 

1 

0 

-1 

-2 

0 

CM12 

2 

1 

0 

-1 

-2 

0 

CM13 

2 

1 

0 

-1 

-2 

0 
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N.  RISK  ASSESSMENT  SECTION 


Question  Number 

Choices 

A 

B 

C 

D 

E 

F 

G 

RA1 

2 

-2 

0 

0 

RA2 

-2 

2 

2 

2 

2 

0 

RA3 

1 

1 

1 

1 

1 

1 

0 

RA4 

2 

-2 

0 

0 

RA5 

2 

2 

2 

0 

RA6 

2 

1 

1 

-2 

RA7 

1 

1 

1 

1 

0 

RA8 

0 

1 

0 

2 

-4 

Choices 

Question  Number 

5 

4 

3 

2 

1 

N/A 

RA9 

2 

1 

0 

-1 

-2 

0 

RA10 

2 

1 

0 

-1 

-2 

0 

RA11 

2 

1 

0 

-1 

-2 

0 

RA12 

2 

1 

0 

-1 

-2 

0 

RA13 

2 

1 

0 

-1 

-2 

0 

RA14 

2 

1 

0 

-1 

-2 

0 

RA15 

2 

1 

0 

-1 

-2 

0 

RA16 

2 

1 

0 

-1 

-2 

0 

RA17 

2 

1 

0 

-1 

-2 

0 

RA18 

2 

1 

0 

-1 

-2 

0 

RA19 

2 

1 

0 

-1 

-2 

0 

RA20  is  to  be  answered  if  a  portion  of  the  system  is  subcontracted. 


Question  Number 

Choices 

A 

B 

C 

D 

RA20 

-4 

2 

2 

2 
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QUALITY  ENGINEERING  SECTION 


Question  Number 

^~Tb~ 

QEl _ 2  -2_ 

_ QE2 _ 2  2_ 

_ QE3 _ 1 _ 1_ 

QE4 _  2  0 

Question  Number  5  4 


Choices 
C  I  D  I  E 


111 


Choices 


QE5 

2 

1 

0 

-1 

-2 

QE6 

2 

1 

0 

-1 

-2 

QE7 

2 

1 

0 

-1 

-2 

QE8 

2 

1 

0 

-1 

-2 

QE9 

2 

1 

0 

-1 

-2 

QE10 

2 

1 

0 

-1 

-2 

QE11 

2 

1 

0 

-1 

-2 

QE12 

2 

1 

0 

-1 

-2 

QE13 

2 

1 

0 

-1 

-2 

QE14 

2 

1 

0 

-1 

-2 

QE15 

2 

1 

0 

-1 

-2 

QE16 

-2 

-1 

0 

1 

2 

QE17 

2 

1 

0 

-1 

-2 

QE18 

2 

1 

0 

-1 

-2 

QE19 

2 

1 

0 

-1 

-2 

Question  Number 

Choices 

A  B  C  D  E 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


380 


APPENDIX  H:  DATA  ANALYSIS  OF  SCORES 


ACRONYMS 

Communication 

C 

People  Area  Score 

PEOPLE 

Teamwork 

T 

Process  Area  Score 

PROCESS 

Leadership 

L 

Product  Area  Score 

PRODUCT 

Organizational  Commitment 

OC 

Risk  Area  Score 

RISK 

Project  Manager 

PM 

PME  Score 

PME 

Stakeholder  Involvement 

SI 

Rounded  PME  Score 

PME-R 

Staffing  and  Hiring 

S 

Project  Success  Rating  by 

PSR 

Requirements  Management 

RM 

Project  Monitoring  and  Control 

PMC 

Project  Planning  and  Estimation 

PPE 

Scope  Management 

SM 

Configuration  Management 

CM 

Quality  Engineering 

QE 

Risk  Assessment 

RA 

Risk  Control 

RC 
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APPENDIX  I:  PROJECT  SUCCESS  POTENTIAL  EVALUATION 
LIST  [FROM  (THE  STANDISH  GROUP,  1995B)] 


Prola-st  Suc-aese  Potential 

User  Involvement 

■  Do  1  have  ire  rlghr  Lser;sJ? 

■  DIb  1  Irvor.e  Che  ueei  &;■  ealy  a.nc  otter? 

■  Do  1  lave  a  qLairy  usehEj  ■eladoiEhp7 
■*  Do  1  na*e  lr'-o -en=rr  easy? 

*  DIB  1  rnd  out  ;xhat  :he  Lser;e)  needs? 

Far  each  qLesIten  wlh  a  YES  answer,  add  2  3  paints 
:a  ire  latal  project  blokes  poemlal  sco-e.  Tate 
=alr:s  iifflo  excess  1 9 1 

-Small  Project  MlleetonBe 

An  1  using  the  3E.  2C  r jIe? 

■»  Am  1  using  a  Lop-do-'.xn  des  gn 7 
■«  At  eel  no  llTe  IItILs7 
■*  An  1  ls  ng  a  pmatyps  tod? 

■*  C3i  Teasuo  progrese? 

For  each  ouestoi  won  a  YES  aiewer  add  1.E 
pair's  la  tie  :cta  crcjeat  success  potertal  saare 
Tol3l  Palrts  matte  exceec  ?'i 

Executive  Management  Support 

*  Da  1  have  the  hey  e*Bouttve»? 

*  Doee  ire  <ey  executive  rave  a  stoke  i  the 
outcome" 

Is  to  lure  acceatocle7 
t  Co  1  have  a’.xel  defnEC  clan? 

■«  Does  tfie  prajEc:  Learn  have  a  Elate? 

=o*  each  qitsBon  wth  a  -ED  anewer,  add  2  2  pants 
:a  ire  latal  project  blokes  po:er:al  score.  Tctal 
=alr:&  no:  lo-  exceec  _6 

Compelenl  Stair 

Do  1  Know  tee  sklls  requi  red" 

Do  1  have  ire  r  grt  neap  e? 

■h  Dd  1  have  a  iaiilng  program? 

«  Do  1  have  ncenUves7 
n  VH  tne  sla"  eee  It  dnraugr  7 

For  e3ch  question  wtn  a  '-ED  answer.  add  1.6 
pair's  la  the  :ata  project  success  po:ir:lal  scare 
Tolsl  Palrts  ;iat1o  Exceec  3j 

Developing  a  Clear  Statement  of  Reqta 
•  Do  1  lave  3  cor  else  vision? 

Col  have  a J  unallaral  3  lalyels? 

■  Co  1  lave  a  nek  assessment? 

-  Col  aave a bus r es i  case? 
t  Can  1  rressu’e  tea  project7 

Fo'  each  question  w:n  a  YES  arswEf.  add  2  pants 
:a  ire  latal  project  BLCoess  pcoemal  score.  Tate 
=atns:s  no:  Lc  exceed  ISj 

Project  Ownership 
■«  Do  1  have  denied  rales7 

Do  1  have  d  def nsc  arganlzat  on 7 
■«  Does  e«eryore  krow  irel-  o  e? 

■*  Ao  IncefiUves  attached  :o  success7 

Is  everyone  canT  ires  7 

For  each  ouestoi  wKn  a  '-'ED  answer  add  1.2 
palate  la  tie  :atai  project  success  po:er:lal  soars 
7oi3l  Palrts  mat  lo  exceed  i'i 

Proper  Planning 

•  Do  1  have  3  arabeT  slstener:7 

•  Do  1  nave  asoluBon  statement7 

■  Do  1  lave  tee  Ugh?  peop e? 

•  Do  1  lave  aim  spectflaaton? 

■  Do  1  lave  shalraPle  mllestores? 

For  eaan  qLesllan  v.-ti  a  YES  aiev.-sr,  add  2  2  palrts 
Id  ire  latal  p’ojEct  blokes  posmlal  sco-e.  Tctel 
Palms  no:  Lo  excess  1 1/ 

Clear  Vie  Ion  and  Objectives 
■«  Is  ire  vlslar  snared? 

■*  s  Tie  vislcr  allgrea  wtte  company 
gaas7 

*  Are  toe  oa|ectves  achevabe? 
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Hard  Working.  Focused  Staff 
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